Polish Voting by Judges

Founder
Posts: 1,138
Joined: 2002.04
Post: #1
I've done my best to gather a range of judges, whom should have a wide range of machines. However, 2 judges, including myself and running G4 Machines at 400MHz. In the event we are unable to launch a game, should the Polish vote be 1, or should the vote simply be removed? Your input is welcomed.

My thoughts is, I'd rather see a message pop up that says, "Sorry, but this game has detected you are running a G3. The minimum requirements for the game are
a G4." And then exit gracefully. Much better than a black screen, and then nothing.

Another example is a game that does not scale well to lower-end machines. If there are settings for, say resolution, textures on/off, etc to help squeeze some extra speed for the game, thereby making it playable, then that would seem like something to reward a dev (in the Polish category.)

Anyhow, the main question -- what to do with games that simply won't run on the Judges machine. (I have suggested a first step to them which is to test said game on a friend's machine, or co-workers.)

Carlos A. Camacho,
Founder
iDevGames
Quote this message in a reply
Member
Posts: 339
Joined: 2004.07
Post: #2
I think a good solution would be to first not count the vote, and then have some sort of flat deduction from the final score.

Obviously if a game doesn't run on a judge's computer it should be penalized, but depending on the size of the panel, just one score of 1 could knock a game completely out of the Polish category.

If you have 4 judges on a scale from 1 to 5, even one score of 1 means a whole point off your final score. If the score instead doesn't count, and you are deducted a more reasonable amount (.1, .25, .5, depending on the size of the panel) an otherwise well polished game still has a chance if it doesn't run on one computer.

It would suck to see flashbacks of Vectorized (my two favorite entries were penalized enough to where they had absolutely no mathematical chance of winning.)

Justin Ficarrotta
http://www.justinfic.com
"It is better to be The Man than to work for The Man." - Alexander Seropian
Quote this message in a reply
Member
Posts: 715
Joined: 2003.04
Post: #3
How about the judge reads the system requirements and instructions posted with the game download?
Quote this message in a reply
Founder
Posts: 1,138
Joined: 2002.04
Post: #4
>How about the judge reads the system requirements and instructions posted with the game >download?
Well, if they did, and the game still doesn't work, then what?

Carlos A. Camacho,
Founder
iDevGames
Quote this message in a reply
Member
Posts: 715
Joined: 2003.04
Post: #5
Camacho Wrote:>How about the judge reads the system requirements and instructions posted with the game >download?
Well, if they did, and the game still doesn't work, then what?

Games should be tested on the judges equipment in advance, and the devs should be informed of issues and machine specs in advance. This issue shouldn't be left to the last minute.

We've had over 5,000 downloads, and about 5 "doesn't run" complaints, do you know how many system specs we've receive with those complaints so that we could trouble shoot the problem? ZERO! Even from devs.
How are we supposed to polish an invisible problem on a random computer?

I don't see in the rules or judging section "games must be playable on 5 year old computers running stock video cards to win"...but I do see that in this thread, not literally but its being hinted at.

Thats like having a Macintosh game dev contest in 1985 and saying "Games must work on Apple II".
For perspective OS 8.5 was released in 1999, same year as the 400mhz G4, but we don't make games that even support OS 9 anymore. Why is hardware any different?

How can a judge honestly review today's games, on the last centuries computers?

Since the standard for the games has been raised, therefore the standard for the minimal equipment of the judges computers should be raised.
Disqualify the judges' computers, not the games.
Respect the work of the developers, not the "appointed" temp job of favored person x.

Think about it, these judges would in effect give a low polish score to HALO, because it won't run on their machines. Would that be a fair vote? Is HALO not polished?
For perspective add to this, TRON 2.0, World of Warcraft, Unreal Tournament 2004, Homeworld 2, Doom III....and just keep rattling off names million dollar games that "ye olde mac" can't play. Guess the collective Billions of dollars they've made means zilch in the eyes of the "Can't play it, give it a low polish score" mentality.

Do the right thing, since you didn't set a minimum system spec at the start of the contest, raise the minimum system spec for judging to the minimum system spec of the games people have made for this contest.
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #6
igame3d Wrote:How are we supposed to polish an invisible problem on a random computer?
As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

If that's not possible (and yes, it is hard), there are some things you can do. For example, if you fail to get a GL context, dump a bunch of system info out and alert the user to email it to you.
Quote this message in a reply
Member
Posts: 201
Joined: 2002.06
Post: #7
Five judges, toss the lowest score and the highest score, average the other three.

(Scale to match total number of judges.)
Quote this message in a reply
Member
Posts: 715
Joined: 2003.04
Post: #8
arekkusu Wrote:As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

If that's not possible (and yes, it is hard), there are some things you can do. For example, if you fail to get a GL context, dump a bunch of system info out and alert the user to email it to you.

Where is the list of "must run"machines for this contest?
Where is the "target hardware"?
Doesn't exist, does it?

Lets make a list based on Carlos' 400mhz G4.
G4/400 (Sawtooth) introduced 1999.10.13
G4/450 introduced 1999.08.31
G4/500 introduced 1999
G4/450 MP introduced 2000.07.19
G4/500 MP introduced 2000.07.19
G4/466 introduced 2001.01.09
G4/533 introduced 2001.01.09
G4/533 dual introduced 2001.01.09
G4/667 available 2001.02
G4/733 available 2001.02
G4/733 introduced 2001.07.18
G4/867 introduced 2001.07.18
G4/800 dual announced 2001.07.18
G4/800 introduced 2002.01.28
G4/933 introduced 2002.01.28
G4/1 GHz dual introduced 2002.01.28
G4/867 MGHz GeForce4 MX introduced 2002.08.13
G4/1.0 GHz dual Radeon 9000 Pro introduced 2002.08.13
G4/1.25 GHz dual Radeon 9000 Pro introduced 2002.08.13
G4/1.0 GHz GeForce4x introduced 2003.01.28
G4/1.25 GHz dual Radeon 9000 introduced 2003.01.28
G4/1.42 GHz dual Radeon 9000 introduced 2003.01.28.13
G4/1.25 GHz single 256/80/Combo introduced 2003.06.23
G4/1.25 GHz dual 256/80/Combo introduced 2003.06.23
1.6 GHz announced 2003.06.23
1.8 GHz announced 2003.06.23
1.8 GHz single introduced 2003.10.19
1.8 GHz dual 512/160 introduced 2003.11.18
2.0 GHz dual 512/160 announced 2003.06.23
1.8 GHz dual 256/80 introduced 2004.06.09
2.0 GHz dual 512/160 introduced 2004.06.09
2.5 GHz dual 512/160 introduced 2004.06.09

Thats 32 different TOWER configurations between 1999 and 2004.
That does not include, imacs, ibooks, emacs, and powerbooks. Or any of the G3 line.

We've personally and succesfully tested this software on 6 of our personal Macs and 2 pc's. One of those macs, is 50mhz below the minimum system spec, and the game ran fine when i followed my own instructions of changing resolution. Why it doesn't work on random machine X which is above spec is still unknown.

If over 5,000 downloads isn't enough "beta testing" I don't know what is.
If fellow developers can't provide adequate system info when reporting a bug, the 1 out of 1,000 regular users who reports a bug obviously can't be expected to do much better.

If "polish" score boils down to 5 macs, 2 of which are 31 revisions from "modern", then the host of the contest should provide the system specs for those 5 Macs as "targets", before people spend time coding or are in a crunch to rewire their work to fit the last minute situation ambush.
Quote this message in a reply
Sage
Posts: 1,232
Joined: 2002.10
Post: #9
igame3d Wrote:Where is the list of "must run"machines for this contest?
Where is the "target hardware"?
Those are two different questions, and I was only responding to the second one. The target hardware is determined individually by each developer according to their game requirements and should be specified in their readme.

The first question is open, I make no attempt to answer it. I agree with you though that the judge's machine specs should be make public in advance. It certainly would have changed my test plan (and likely, score results) in the Vectorized contest if I had known in advance that Brad Oliver would be testing with a dual-card-dual-monitor setup.
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #10
arekkusu Wrote:As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)


Personally, I've had a hard time getting any feedback at all from over one thousand downloads. Not getting bug reports doesn't mean that the game worked fine for everyone, just that I won't know about it = won't know it needs fixing!

I tried my best to make sure my entry ran even on a Rage 128 Pro (16MB), but that's the bare minimum and really on the edge, not only for my entry, but for practically any OpenGL based game.
Quote this message in a reply
Moderator
Posts: 916
Joined: 2002.10
Post: #11
best beta testing, releasing it to the IMG forums and say "hey guys, bitch at me".

seriously, they are great for feedback
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #12
My entry's final version is up, feedback from you guys would be really appreciated!
BTW, Carlos, did the Judges vote already? If not, did they download all the entries at the beginning of the contest, a few days ago... yesterday? Also, are they using the old download page or the new system (I haven't updated my info there yet)
Quote this message in a reply
Founder
Posts: 1,138
Joined: 2002.04
Post: #13
>BTW, Carlos, did the Judges vote already?
Only 3 have voted thus far. I've held off in asking them during the first week. Since the Poliosh category works the same way as the other categories, understand it isn't the amount of votes that you get. (Just wanted that to be made clear.)

Anyhow, please come back to a bit of reality. In the start of the contest, I have a list of say 10 people to be final judges. These people say yes. No contract is signed of course. I could post their spec machines, but.....

Now we get to the end of the contest. That list of 10 is down to 3. What can I do? Sue them for being busy? They are after all, people who work fulltime in this business. (Read the EA thread to see what that is like.) Yes, in a perfect world, all 10 judges would have brand new machines, and they would indeed see the contest out until the end. So, that is basically why I don't mention anything about judges during the contest. Lucky for me, I know ahead of time that most likely we will have judges that go AWOL, or whose commercial projects are behind schedule, etc, so I have a backup list of judges who can assist this community. Those people have been contacted and are chipping in. I'm extremely confident that they will give this community their best effort in helping us recognize the best Polished games.

So to rephrase, would it have made a big difference if you got a list of say 30 different machines (all 1st judges and backup judges) at the start of the contest? Although devs usually run fast and latest machines, my guess is that the list of machines would be similiar to a random sampling of people in this community or gamers. Would knowing that Carlos, with his old machine, 'cuz he is too poor to buy a new machine, due to iDevGames Wink have people at the start of the contest saying, "I'd like to have Judge CC removed." Kind of selecting the jury. Grin

Cheers

Not sure if I am explaining myself right. I think that arekkusu has a good grasp of it though.

Sorry if that's not really the reply everyone is looking for... a bit tough trying to juggle so much. I do agree for Mini-contests, where the time frame is so short, posting judges Mac specs is do-able. Since all 4 judges that were asked, came through at the end.

Cheers,

Carlos A. Camacho,
Founder
iDevGames
Quote this message in a reply
Moderator
Posts: 130
Joined: 2002.04
Post: #14
So, igame3d, are you saying that a game that runs on more computers(but only Macs, since this is a Macintosh game contest) shouldn't get rewarded?

And yes, I think the Mac version of Halo should get knocked down a notch becouse of it's high system requirements and low possibilites to change that(ie, make the graphics look uglier).

"Gameplay Uber Alles. And if you can make it psychedelic too, great!" - Jeff Minter
Quote this message in a reply
Member
Posts: 184
Joined: 2004.07
Post: #15
arekkusu Wrote:As a developer releasing software into the wild, it is your responsibility to test it on all of your target hardware. That means getting access to a lab, or a lot more beta testing (with a list of "must run" machines, checking off each config as you get testers.)

If that's not possible (and yes, it is hard), there are some things you can do. For example, if you fail to get a GL context, dump a bunch of system info out and alert the user to email it to you.

I think the problem with the latter part of what you're saying is that at this point, it's too late. Possibly, the judge could send you this information, you fix it, judge runs again, but people really don't have time for this if it's gotten to this point.

For a professional piece of software, yes, it's very sloppy to not test on a huge lab with many different configurations. You can probably rent time out of one of these, though obviously for a contest of this scale, it's not as if any of us are at liberty to do that. Even if the judges post their specs, it doesn't mean any of us might have access to said machine, so that probably won't help (if you didn't even know about the dual-monitor bug, it wouldn't have helped to have been told a judge had a dual-monitor setup.)

I agree with Justin- if a judge that can't run the program, don't consider his vote, but give some deduction per each judge that can't run the program. The amount of penalization is up to you, though one easy soution might be -25% off your total for each failed computer.


From my personal experience for this contest, by the way, I've only received a few comments about crashing, and for the majority of them, I've either been ignored when asked for the debug files that would tell me what was wrong, or the person who crashed was anonymous, and I couldn't contact them. So even trying to ask the community for help is no guarantee you will get anything. Again, I believe an entry for a contest of this scale should be graded differently performance-wise than professional software.
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  uDG08 Voting Results and Methods AnotherJake 111 47,065 Mar 20, 2009 05:15 PM
Last Post: pinguoren
  uDG voting DoG 11 5,735 Mar 7, 2009 09:57 AM
Last Post: FlamingHairball
  Questions regarding voting? FlamingHairball 20 9,129 Mar 6, 2009 10:00 AM
Last Post: FlamingHairball
  How will public voting work? pinguoren 7 5,025 Mar 2, 2009 11:22 AM
Last Post: Carlos Camacho
  Networking Minicontest "Voting" aarku 26 10,791 May 9, 2008 08:36 PM
Last Post: PowerMacX