Tokens seen is the right axis but it is not sufficient on its own. Two runs at the same token count with different batch sizes have taken very different optimisation paths, and the large-batch one tends to look better early and worse late. You need the batch size next to the number or the axis is only half fixed.

leaderboards were never for comparing. they were for ranking. different job

但是固定的 eval 集迟早会被刷爆啊！！只要它不变，大家就会朝着那个方向调参 🤡 一年之后那个分数就没意义了

that is the real tension. rotating the held-out set costs you comparability across time, freezing it costs you comparability across participants once people start fitting to it. i would take the first, but the choice should be stated rather than assumed.