uDG08 Voting Results and Methods

Moderator
Posts: 625
Joined: 2007.09
Post: #91
Alrighty!

tcIgnatius Wrote:Here's a situation to help illustrate:
10 people vote for a game in the category most original.
8 give a score of 9
1 gives a score of 7
1 gives a score of 1

The mean/average gives this game a score of 8. We're all good.

Quote:Now, imagine those people discussing the vote. The mean would say that game got an 8 in originality. Everybody who voted 9 is a little confused as to why this is. One of the voters who voted 9 goes around the room and asks others how they voted. He asks the seven other votes for 9 and still wonders how this game got an 8. He talks to the person who gave it a 7. The person who gave it a 7 claims that its not all that original, the seven who voted 9 agree this is an original concept and the one who voted 7 admits to being a bit more critical on the topic of originality. The 9 voters then say "well, ok, so it sounds like the mean should have been 8.7", although the majority thought it was better than that. Now they look at the person who voted 1. "Why did you give this game a 1? This isn't a remake or anything close to that." The 1 person explains any of the following:

There are four options here:

Quote:it was an error
I didn't think it was original
I didn't play it.
I tried to strategically sink it.

I'll go with the second one; the others seem to be attempts at humorous sarcasm.

This guy thinks the game is not original. Tell me: why does his vote count *less* than the others?
He played the game, he's entitled to his opinion just as much as the other 8 guys.

Quote:You see it doesn't really matter. The game didn't deserve an 8, it deserved a 9. A review should reflect the majority opinion. When we have so few votes, it's really not fair that one person can change the score. (and have more weight the further it deviates from the median value)

Yes, it is fair. It did deserve an 8. See above..
No, the review should reflect the opinions of the voters.

Quote:If you do not see my point now, You are still stuck on thinking that the mean is more fair, only because it does intuitively seem like it would be.

Only, because it intuitively seems like it would be?
The mean is more fair because it takes into account the opinions of the whole group. Prove that statement wrong.

Quote:Please assume that it is not fair, then come up with a contradiction. This is how you prove something. It's called proof by "modus tollens" for any logic geek. If you try to prove that the mean is fair through an example, you should realize that there is no proof in that. Proof by example is also called "no proof at all".

Look who knows their latin. See, what you just asked me for is an indirect proof(there is a difference).
I like to think that I've remained fairly calm throughout the course of this thread, but the deferential tone that pervades your posts has begun to annoy me.
I haven't seen you prove your point.

And, it's a bit hard to prove my point, since we seem to have different definitions of 'fair'.

- Lincoln Green
http://www.binkworks.com/
Moderator
Posts: 3,591
Joined: 2003.06
Post: #92
Well, look, what he's saying is that there are in fact voting irregularities that show up in these contests (whatever those irregularities may be: strategic voting, voter error, voting system error, etc.), just like any other voting situation. Most of us say, well, we're all exposed to the same chance of irregularities (which can be argued as true), and we think that's fair. tcIgnatius is arguing that those irregularities can be effectively filtered out, and if you don't make an effort to filter them out he feels it's unfair, which can also be argued as true. Other systems make attempts at filtering the results. This one could too. Most of us don't want to. He does. That's the bottom line. That's what we've been discussing for nearly 100 posts in less than 24 hours.

It's my opinion that this is about as close as we'll get to an agreement here.
Member
Posts: 65
Joined: 2008.09
Post: #93
Quote:The mean/average gives this game a score of 8. We're all good.
You once again show that you fail at seeing any other point of view.

Quote:I'll go with the second one; the others seem to be attempts at humorous sarcasm.
You assume one possibility, and stick to it. This is your problem. You would benefit from seeing all possibilities as options.

Quote:This guy thinks the game is not original. Tell me: why does his vote count *less* than the others?
It DOES NOT count less. Each vote gets EQUAL representation. Under the mean, a vote that deviates from the median counts MORE.

Quote:Look who knows their latin. See, what you just asked me for is an indirect proof(there is a difference).
Yes, and you only knew that because you had to look up what modus tollens is. The two forms are interchangeable in some logical proofs, which is why I tend to get them mixed up. Both will result in a valid argument though. (something you neglected to mention)

Quote:And, it's a bit hard to prove my point, since we seem to have different definitions of 'fair'.
Moderator
Posts: 776
Joined: 2003.04
Post: #94
I think the problem is that by discussing whether the mean, median, etc should be used we are assuming two things that are not necessarily true:
- the scores have the same meaning for everyone: not true, I can consider 7 a very good score whereas someone else could consider it merely "good"
- a small minority assigning an odd score probably means something suspicious is going on: again, not necessarily true. A couple of examples: a game may have graphic problems in some machines so the players may justifiably score it lower in graphics/polish, or they may have previously played some obscure (or just very old) game that is very similar and assign an entry a lower originality score that most others

I think that the focus should be simply in increasing the number of people who bother to rate the entries. Scoring 20+ games in a scale from 1 to 10 trying to be consistent is difficult and time consuming. I think a simple grid with a small screenshot (*) of every game, the title and a one line description, followed by a simple "1-4 stars" (**) rating control below would encourage a larger fraction of the general public who downloaded and played the games to actually vote. The loss in precision going to a 1-4 stars (vs. 1-10) would then be compensated by the large sample of players.

* (sometimes its hard to remember which entry is which just going by the title if you played a lot of new games in a matter of a few days)
** (I say 1-4 instead of 1-5 to force people to decide whether a game is good (3) or bad (2), otherwise many would simply vote "average")
Moderator
Posts: 1,563
Joined: 2003.10
Post: #95
Unacceptable way of writing this:
tcIgnatius Wrote:You once again show that you fail at seeing any other point of view.

Acceptable way of writing this:
Quote:I don't think we're seeing this from the same point of view. [explanation of differences here]

------

Unacceptable way of writing this:
tcIgnatius Wrote:You assume one possibility, and stick to it. This is your problem. You would benefit from seeing all possibilities as options.

Acceptable way of writing this:
Quote:They were all meant to be valid possibilities. Consider the problem from the viewpoint of all four.

------

Knock it off or I lock the thread. The exact same argument substance in a neutral tone is perfectly OK. Writing it as inflammatorily as you have been is not.
Moderator
Posts: 869
Joined: 2003.01
Post: #96
A truncated mean is the simplest method for reducing personal bias. The median, and any other method that requires you to pick a number from a set, assumes particular distributions which aren't necessarily representative, and impossible to figure out without effectively knowing the intent and behavior of the voters, which we don't.
Moderator
Posts: 869
Joined: 2003.01
Post: #97
PowerMacX Wrote:I think that the focus should be simply in increasing the number of people who bother to rate the entries. Scoring 20+ games in a scale from 1 to 10 trying to be consistent is difficult and time consuming. I think a simple grid with a small screenshot (*) of every game, the title and a one line description, followed by a simple "1-4 stars" (**) rating control below would encourage a larger fraction of the general public who downloaded and played the games to actually vote. The loss in precision going to a 1-4 stars (vs. 1-10) would then be compensated by the large sample of players.

I'd just like to point out that this suggestion is purely a technical issue --- an admittedly important one. The voting software was simply crap. Every download page should have also linked to the appropriate voting page, if not have had it inline, and summaries would've been great. (Just as a note, I boycotted the peer voting because of all the issues with the voting software, which were all ignored, and for the public voting, I haven't voted for even half the games I wanted to, simply because it was, indeed, such a hassle)
Moderator
Posts: 3,591
Joined: 2003.06
Post: #98
DoG Wrote:A truncated mean is the simplest method for reducing personal bias. The median, and any other method that requires you to pick a number from a set, assumes particular distributions which aren't necessarily representative, and impossible to figure out without effectively knowing the intent and behavior of the voters, which we don't.

Exactly my point of view! We don't know the intent, nor do we know why it happened. It literally could have been caused by voting malfunction and had nothing to do with any intent. We'd like to trim out the blips, but I don't think we'd want to go so far as removing representation altogether, which is what any median method winds up doing to at least some extent (even truncation, although not necessarily to the same extent). It's a balancing act between having all the votes being *justifiably* represented, and the votes being *fairly* represented. If we try to make them all "fair" by heavy filtering, then we create a situation where representation is literally removed from some (potentially *most*) folks. If we let everyone's vote count, even if they say it was a mistaken vote, then it's not necessarily fair and we don't want that either. Balance is best. As has been stated several times already, the truncated mean is used in many situations in the real world for voting purposes. After all the discussion in this thread, I see why now.
Member
Posts: 37
Joined: 2009.02
Post: #99
Long live the truncated mean!
I'd say for this kind of judging system it offers the best solution.
Member
Posts: 65
Joined: 2008.09
Post: #100
Quote:Knock it off or I lock the thread. The exact same argument substance in a neutral tone is perfectly OK. Writing it as inflammatorily as you have been is not.
I apologize for my tone. With all due respect I feel a bit slighted because other people lash out at me and there's no moderator intervention.

Quote:Long live the truncated mean!
I'd say for this kind of judging system it offers the best solution.
Yes the truncated mean is good, but I think the interquartile mean (which happens to be the best variation on the truncated mean) would be the best of the compromised solutions. As long as it's not just the mean of the scores I think it should be fair.

As a side note: This whole argument should have no effect on the current contest results nor was it ever my intention to change the results. I think a lot of people are showing me hostility because they think I'm trying to change the outcome of the contest. I only want to supply next year with a better way to score the games.
Moderator
Posts: 3,591
Joined: 2003.06
Post: #101
tcIgnatius Wrote:I apologize for my tone. With all due respect I feel a bit slighted because other people lash out at me and there's no moderator intervention.

I don't know if you've noticed it, but even though I have my own opinion on this [which you did manage to sway somewhat, and enlighten me a bit], I've been trying to watch your back here. The moment you said you felt like you needed to stick up for some justice, I smelled a new thread. Your voice will always be heard here as long as the tone doesn't get nasty. This is a tough group to deal with at times, and I think you deserve a pat on the back [not meaning to sound condescending in any way with this] for being "the new guy" with an opposing opinion, and sticking through this thread and making your argument without getting out of line. You certainly have my respect.
Member
Posts: 156
Joined: 2002.10
Post: #102
As we seem to have come to a consensus here (on some level of truncated mean), I think we need a moderated thread to store 'community-agreed' changes that we want to make for uDG 09. There are probably more suggestions in the uDG 09 Discussion thread, but I think it is worth pulling out the key agreed points into simple bullet points so that we can give them to Carlos to think about implementing (or some of us implementing, if possible).

Cheers

- Iain
Member
Posts: 44
Joined: 2008.12
Post: #103
diordna Wrote:USA example: Would you throw out a Republican vote in a strongly Democratic district just because it was an "outlier"? Come on now!

Actually we kind of do do this. That is what the electoral system effectively does.
Member
Posts: 245
Joined: 2005.11
Post: #104
PowerMacX Wrote:I think a simple grid with a small screenshot (*) of every game, the title and a one line description, followed by a simple "1-4 stars" (**) rating control below would encourage a larger fraction of the general public who downloaded and played the games to actually vote. The loss in precision going to a 1-4 stars (vs. 1-10) would then be compensated by the large sample of players.

I like the idea of having a big list of all the games on one page (with screenshots and links) and being able to vote for them all there. It would help if people could come back to this page and amend or add more votes as the play more games. And there would need to be a "haven't played it yet" option.
The current system is kind of slow and clunky. When I came to vote I had a notepad with notes on all the games (except the ones I couldn't play) and it took me quite a long time to enter scores for the 15 or so games.
On a related note, actually downloading everything in the first place could have been much quicker and easier. A downloader script or app would save a lot of clicking.

I don't want to restart the argument but I'd say that a truncated mean makes sense to me. I do agree that there may be instances where one guy gives a game a very different score for a very good reason though (maybe he ran into a bug nobody else experienced, or perhaps he just "got" the gameplay when nobody else really understood it). And it still doesn't deal with the guy who spends four days acquiring new IP addresses and voting for himself again (it could happen!).
Member
Posts: 65
Joined: 2008.09
Post: #105
Quote:This is a tough group to deal with at times, and I think you deserve a pat on the back [not meaning to sound condescending in any way with this] for being "the new guy" with an opposing opinion, and sticking through this thread and making your argument without getting out of line.
I wonder if using one of my old avatars (zerocool, icecool, ignatiusIce) would have made any difference.

Quote:You certainly have my respect.

You certainly deserve my thanks for putting up with me.

Quote:I think we need a moderated thread to store 'community-agreed' changes that we want to make for uDG 09.
I second this notion.