Player-generated content woes

[shasarak]

So you don't discard anything until all votes have been cast? Okay, that makes sense.

[KaVir]

Well this is just my interpretation of MECHFrost's approach, I'm not quite sure if I understood it correctly, but I think he's saying it works like this:

1. Add the new vote to the full list.

2. Calculate the average vote of the full list.

3. Initialise the new smooth list as an identical copy of the full list.

4. Discard all extreme votes from the smooth list (but ONLY from this list).

5. The final rating is the average vote of the smooth list.

So the full list never discards any votes - some votes may not be included in the current final rating, but they're still stored, and may be included later if the rating rises or falls far enough (as more votes could be added at any time).

Thus in the example you gave, where a load of bad votes are given early on, the rating would initially be very bad - and good votes would be ignored as extreme values. But as the number of good votes increased (assuming there were no more bad votes coming in) the average (in step 2) would gradually increase, and eventually it would reach the point where the bad votes were considered "extreme" and all of the good votes were counted.

This could cause a bit of a yo-yo effect though...perhaps I've misunderstood the suggestion.

[MECHFrost]

Yes, this is a correct, but since the average calculated in step 2 is not the correct average, steps 2 to 5 need to be repeated several times, each time with a new average that's going up to its true value. Otherwise high votes that are correct might get discarded as they are too far from the first (incorrect) average calculated. From my tests, they would only need to be repeated once unless the level of "corruption" is high.

Today, I calculated how many votes it must accept before displaying accurate results. My calculations are based on statistics so they are not themselves perfectly accurate, but they give a good idea.

For each level of corruption, I generated 1000 lists of votes with a random average between 60% and 100%, and I checked how many votes it takes for the average to be stable (i.e. within a decently close distance of the true average) for 100 turns/votes (changing that value didn't change the result; once it's stable for a few turns, it's stable forever - with a fixed amount of corruption).

The following screenshots show the results for:

1) how many votes on average it takes for the result to be stable (decently low)
2) how many votes at max it took for the result to be stable (can be very high, with bad luck) - result 2) is not accurate at all since the average is calculated out of 1000 samples whilst the max is only calculated from 1
3-4) the same number, but with a reliability %, just to discard the "exceptional" results, i.e. what is the number of votes required for a certain % of the 1000 lists to be stable.

Screenshot with a distance of 39:

http://img413.imageshack.us/img413/5907/distance3.jpg

e.g. with a corruption of 30%, it takes ~52 votes to have a 97.5% chance of stability.

The results are better if the average is between 80% and 100% as it's the range where the distance 39 works best.

http://img686.imageshack.us/img686/700/screenshot4e.jpg

[KaVir]

Going back to one of the original points:

[MECHFrost]

One problem that I can see is that the bots will rate the difficulty as too high. An example is if the players always run invisible past monsters and don't fight them unless they have to, the bot might not do the same and fight all the monsters on his path, rating the area as difficult whilst all the players have to do is run invisible past the monsters and get the treasure from a chest at the end. Or more generally and pertinent, if there's a multi-path options (or similarly, dialogues with NPCs), one easy path and one difficult: depending on the path that the bot chooses, it will give a rating that doesn't reflect the combination of the two paths. Unless the bot divides itself each times it has a decision to make, and evaluates every option. The bot can always be refined, but will it be able to keep up with the new tactics that the players come up with?

This is an interesting idea though. The bot would have to be compared to the players' rating to see which one is the most reliable.

[ide]

It seems like you would want to make some of the bots influence the difficulty ranking more than others; for example, an invisible bot dying should add more to the rank than a vanilla tank dying.

[KaVir]

[Kernal]

Assuming that you want player-generated content at all levels, then having a few static bots using preset strategies probably won't work. The hard dungeons will all be 0; the lowbie dungeons will all be 10s. I propose the following edition:

Whenever a bot dies, it increases its level (in some preset manner) before re-entering the dungeon. Its rating then becomes the lowest level at which it completes the dungeon. The dungeon difficulty is then the average of the minimum level required for completion for each different strategy bot. This also allows sensibly categoried difficulties, as MechFROST suggests above.

Unfortunately, this will probably only work for straightforward hack & slash dungeons. Bots will be unable to gauge the difficulty of a dungeon in which there is, for example, a puzzle (assume a puzzle whose solution is not identical in each instance). If bots are given free passes through puzzles, then the rating is instead thrown off for any puzzle which involves some combat (or other form of resource use).

I am a big fan of ratings (difficulty, for example) which are consistent. Even if they're imperfect, very much can be said for consistency, so this bot scheme sounds delightful (puzzles excluded).

Cheers,
Kernal

[KaVir]