Short answer: No
We deliberately chose not to reset the ratings to 1500, and think this will give better results. The glicko2 system is extremely good at rating players, and problems are introduced only when we deviate from the ‘standard’ system glicko is expected to function on.
The main concern we have in general with placing people is the fact that algos cannot rematch eachother. With this consideration in mind, we can predict the outcomes of resetting/not resetting the leaderboard:
Not Resetting algo rating:
Algos should start reasonably close to their ‘true’ rating. They will match against other algos, oscillating around their ‘true’ rating and get closer, on average, after each match.
Resetting to 1500:
Algos start at 1500. They will move towards their ‘true’ rating, then oscillate around it, getting closer, on average, after each match.
Where does the problem come in? What I am concerned might happen is the top algos will take an early lead and move up to around 1700 for example. Then they will match against eachother more often than not as they continue to climb up on average. By the time they are all close to their proper placement, many of them will no longer be eligible to face each other. For a majority of algos, this is not a problem, but some algos will ‘waste’ their best matchups or get through their worst matchups while climbing. With 100s of algos, the most lucky of them would be likely to be bumped up a few places.
I haven’t crunched the numbers, but the alternative seems much better in my opinion: Starting value matters very little in glicko compared to the potential to enter into a ‘favorable pool’ at the top after getting lucky (bad) matchups earlier on.
To address KauffK’s point more specifically: As long as they play a solid sample of algos the pool of top algos, the ratings should sort everyone pretty well within that group.
Monday next week at an unspecified time (We need to do a deploy to undo the hack we did that allowed S3 algos to match for a bit) games are going to stop being played and we will run the competition. The stream will be Tuesday, September 17th at 6pm
I’m sure there will be a few people interested in looking into glicko2, this is the best resource to start with.
If you are interested in rating systems in general, Microsoft’s Trueskill was interesting to read about as well