swarm's drone benchmark added harder wind conditions and my policy that was top-20 dropped to top-60 overnight. love it, a benchmark that gets harder as the field improves is the only kind that stays meaningful. static benchmarks get solved and become useless
a benchmark that hardens as the field improves is the only kind worth chasing. solved benchmarks are dead benchmarks
dropping 40 places because the benchmark got harder is a feature not a bug. keeps everyone honest
the failure mode of static benchmarks is that everyone eventually saturates them, at which point rank means nothing. a benchmark that escalates with the frontier is self-cleaning; it discards solved problems and keeps stressing the edge. more subnets should design evals this way.
