Isaí Roberto Sotarriva Álvarez

fastelo — a chess bot that plays at your level and rates you as you play

A rating system needs dozens of games to place a new player. fastelo tries to do it in one or a few: you play a timed game against a bot that plays like a human of your level, and after every move the same network works out at which rating your moves are most likely, with its uncertainty. The demo at the top of this page is the real thing; the report below it has every measurement.

How it works

The player is a 7M-parameter transformer that predicts the move a human of a given rating would play. It first learned chess by imitating the Leela Chess Zero engine on 300 million positions taken from human games, and was then trained on 500 million human moves from Lichess rapid and classical games, conditioned on both players' ratings.
Calibration, without training. Predicting human moves is not the same as playing at the right strength. On 30,000 positions analysed with Stockfish, the temperature, the share of a stronger engine-distilled network and a small correction of the rating input were fitted so that the bot gives away the same winning chances per move as humans of each rating. It matches them from 600 to about 2400.
Your rating, from the same network. The player gives the probability of every move at every rating. Summed over the moves you played, that is a curve over ratings; its centre is the estimate and its width the uncertainty. It uses only your own decisions, so it does not matter how well the bot plays. And because each move is scored on its own, the curve shows where in a game you played above or below your level.

Error bars that mean something

Early in a game the curve is wide and its centre sits near the middle of the scale whoever is playing, so weak players are rated too high and strong players too low. I treated it as a detector response: measure the bias on held-out human games with the same number of players in every rating band, correct it with a map that is unbiased by construction, and quote what is left on an independent set as a systematic error, separate from the statistical one. The statistical error shrinks as you play more games; the systematic one does not. One game of 20 moves gives about ±300 Elo; five games bring the typical error to roughly 60 to 150. Ratings are only quoted between 600 and 2200, the range where the models were validated.

A separate rating network, tried first

The original design had a second network that read the whole game and predicted both players' ratings. It looked accurate on human games, but mostly because human opponents are matched: it had learned to read the level of the game rather than the player. Against a bot whose strength follows the estimate, that is circular, and the rating drifted; a user playing an engine's best moves was rated below 700. Reading the rating from the player's own move probabilities turned out to be at least as accurate on real players and does not depend on the opponent, so the second network was dropped. The report shows the measurement.

From the weekend prototype

The first version of this project was a single notebook with toy models, a separate critic network and a PID controller steering the bot. This version is a tested Python package; the critic and the controller are gone (the bot simply re-targets your current best estimate before each move), training took about 33 hours on a single RTX 3090, and the demo runs in a small serverless function without a GPU.

Links