Who would have thought computer game developers were such nerds?

We should work out the details for this where it will stay relevant for a year:
uDevGames 2009 Discussion Thread

Let the winners have their moment of glory and pay respect to their worthy adversaries.

IBethune Wrote:I picked up all the results off the uDG website and stuck them in a spreadsheet which is here: http://www.pyramid-productions.net/downl...esults.xls. I've also put in the ranking in each category which makes it easier to compare. I also added in the number of prizes each entry was eligible.

*** IGNORE THIS POST; INFORMATION BELOW IS INCORRECT ***
*** IGNORE THIS POST; INFORMATION BELOW IS INCORRECT ***
*** IGNORE THIS POST; INFORMATION BELOW IS INCORRECT ***

Just a quick note about the prizes column:

My entry (Maneuv'It!) is not eligible for 3 prizes but only 2, because my two 3rd-place-finishes are in the same "prize-round" and the rules state that more than one choice per round is not allowed. So I get to pick once in the 3rd round, and once in the 4th round.

As another example Laserface is not eligible for 6 prizes, but rather 3. One pick in each the 1st (for best overall), 2nd (for its four 1st place finishes) and 4th (for its one 3rd place finish) rounds.

tcIgnatius Wrote:I was more inclined to forget this, but I just watched flash of genius and I'm in a justice bringing mood.

I disagree with you in that a couple damaging votes don't play an important role. It could mean whole integer differences in some cases and some entrants were decimal points away from winning something.

Yes, there were a few whole integer differences when I tested out the possibility of dropping abnormally low votes. But as I said, those tests did not change the outcome of the contest for the top three, so I concluded my audit at that point, satisfied that even if there were a dispute about this, the outcome would stand justified regardless. Still, I think this is completely valid discussion, so let's carry on a bit:

tcIgnatius Wrote:Same goes for nerfed votes that are positive for the entrant. I just think a mean average would suite this competition better because it would account for nerfed votes.

ex:
p1 scores: 3,5,5,5,6,6,6,7,7,7,7,7,7,7,7,7,7,8,9=6.47
p2 scores: 5,5,5,5,5,6,6,6,6,6,6,7,7,7,8,8,8,8,9=6.47
Is that really fair?
p1 gets nerfed with a 3, and five of p2's comrades "overscore" him.
didn't p1 deserve a 7, and p2 deserve a 6?

Personally, I don't think so. I think the average result is fair. Let me demonstrate, using your example:

p1 scores: 3,5,5,5,6,6,6,7,7,7,7,7,7,7,7,7,7,8,9=6.47
p2 scores: 5,5,5,5,5,6,6,6,6,6,6,7,7,7,8,8,8,8,9=6.47

You have a point that a judge might vote real low -- and it did happen in this contest, in more than one instance. I should remind you that it's still a vote though: Not everyobody likes everybody else, and not everyone is in their right mind when they press the submit button. What if it was merely a mistake? Voting equipment malfunction, voter error, and hanging chad come to mind

So let's say we arbitrarily add a bit of fairness and drop the lowest votes. Remember, if you do it for one, you'd have to do it for all to be fair. When I fiddled with the votes to test things, I didn't even do it for all, which further should have skewed the low vote higher. In this case we will be fully fair and drop both:

p1 scores: 5,5,5,6,6,6,7,7,7,7,7,7,7,7,7,7,8,9= 6.67
p2 scores: 5,5,5,5,6,6,6,6,6,6,7,7,7,8,8,8,8,9=6.56

You're right, PLAYER ONE now clearly wins! Justice is served!

But where do we stop? Should we then go ahead and say we'll just take the lowest ten off of everyone's vote? Let's try it:

p1 scores: 7,7,7,7,7,7,7,7,8,9=7.3
p2 scores: 6,6,7,7,7,8,8,8,8,9=7.4

Now PLAYER TWO clearly wins! As you can see, it isn't so easy to call it justice anymore... Herein lies the problem. If we had used a [median], would those who then lost because of using that method then have a right to demand we use average for the next contest?

While I agree that it is valid to discuss the possibilities of being unfairly voted for, that doesn't change the validity of the current scores in any way in my opinion.

[edit] Darn it, you got me saying "mean" too, when I meant to say "median". I knew what you were talking about though

Quote:But where do we stop? Should we then go ahead and say we'll just take the lowest ten off of everyone's vote? Let's try it:

p1 scores: 7,7,7,7,7,7,7,7,8,9=7.3
p2 scores: 6,6,7,7,7,8,8,8,8,9=7.4

Now PLAYER TWO clearly wins! As you can see, it isn't so easy to call it justice anymore... Herein lies the problem. If we had used a [median], would those who then lost because of using that method then have a right to demand we use average for the next contest?

This is not the equivalent to what I was doing. Dropping out low scores then taking the mean does not equal the median. In the above example you've made it so that p2's nerfed votes now count even more than they had before.

I'm going to split off the voting discussion to another thread as soon as I can figure out what goes where.

Quote:If we had used a [median], would those who then lost because of using that method then have a right to demand we use average for the next contest?

No, i think those who won, would've still won, unless there was serious nerfage.

All I ask is that next year we take the median vote. In the case of a tie, then you should use the mean of all the categories.

example:
Game A gets an 8 in graphics
Game B gets an 8 in graphics
Game C " "

Game A has a mean score for all categories equal to 7.28
Game B has a mean score for all categories equal to 7
Game C has a mean score for all categories equal to 8.54

We conclude that graphically Game C is 1st, A is 2nd, and B is 3rd.

There, I hope I got everything moved to the right thread. Pipe up if I moved a post in error!

tcIgnatius Wrote:This is not the equivalent to what I was doing. Dropping out low scores then taking the mean does not equal the median.

I wasn't trying to demonstrate median (I know what the median is, BTW), I was trying to demonstrate that fiddling with the vote to inject fairness doesn't lead anywhere except to unfairness.

My statement about using a median average is tied to the same concept -- one way or another it will bias the vote.

I don't personally feel that median votes are somehow more fair or representative of what a game deserves. But that's just my opinion. There are raging debates about using medians in voting all over the place, so this is by no means a new debate

Ideally, you could drop the top and bottom 10 or 20%, like with gymnastics and similar sports. There, the judges are, particularly in small contests, also the coaches or teammates of some of the competitors, so the top and bottom scores from the 5 or 7 judges are dropped to eliminate such bias.

I actually think that cutting off the top 10% and bottom 10% is a good idea, but that would obviously have to be established before the contest.

Quote:My statement about using a median average is tied to the same concept -- one way or another it will bias the vote.

I don't personally feel that median votes are somehow more fair or representative of what a game deserves. But that's just my opinion. There are raging debates about using medians in voting all over the place, so this is by no means a new debate

In this situation a median vote is fair, there is no logic that would prove otherwise.
Even in a special case:
p1: 6,6,6,6,6,6,6,6,6,6,8,8,8,8,8,8,8,8,8,8
ten 6's and ten 8's
It should be said that in this case, the median will equal the mean at 7.

I've proved than a mean is unfair, yet I cannot prove the median is unfair. Therefore, I can conclude that a median would be fair, regardless of personal opinion

Quote:I actually think that cutting off the top 10% and bottom 10% is a good idea, but that would obviously have to be established before the contest.

That would be more fair, but still imperfect. Even in that first example this method gives p1:6.2 and p2:6.4, those nerfed votes were still the reason p2 won.

I think that it looks like things turned out fairly, but having run a peer voting competition before, I thought I'd offer an observation. If you aren't careful, the system can favor harsher critics, and penalize generous ones.

There is nothing wrong with being a harsh critic or a generous critic. Everyone votes differently, and as long as they vote consistently and fairly, that should be ok.

The problem is that people don't get to score their own entry. Because of this a harsh critic that consistently scores everything slightly below average, can unintentionally raise their own score by lowering everyone else's. Conversely, a generous critic who score's everyone consistently a point above average hurts their own score by raising everyone elses.

In the competition I ran, I decided to add a dummy score for their own entry to everyone's average. This dummy score was computed as the average of the scores that they gave everyone else in that category. This meant that a harsh critic would have their harshness factored into their own score, and a generous one likewise.

The system isn't perfect, and can still be gamed but it at least helps unintentional skewing of the scores based on how people naturally judge things. The larger the pool of contestants, the less its an issue. I had about 8 in the one I ran, and found it made a big difference.

I don't know if this was considered during scoring. I tried very hard to ignore this knowledge and vote naturally during the contest.

I'd be inclined to just leave it the way it is. I think the average makes more sense than the median, and no matter what you do there will by ways to cheat.

Now it may be a good idea to rate your own entries, but again...

AnotherJake Wrote:I actually think that cutting off the top 10% and bottom 10% is a good idea, but that would obviously have to be established before the contest.

I agree with this and would prefer to see this system in the future if peer or public voting is used again.

Quote:I'd be inclined to just leave it the way it is.

Why, if it caters toward nerfing scores?

Quote:I think the average makes more sense than the median

How?

Quote: and no matter what you do there will by ways to cheat.

A median score would almost guarantee no nerfing. Whereas a mean condones it. Give me a counter example to my argument and I'll agree with you.