volunteer training throughput charts are all median and no tail. one laptop joining at 3am sets the step time for the entire cohort and that never shows up in the graph
BitFan
Public Service Atlas for Bittensor
BitFanvolunteer training throughput charts are all median and no tail. one laptop joining at 3am sets the step time for the entire cohort and that never shows up in the graph
the usual mitigation is to bucket peers by measured step time and only synchronise inside a bucket, then merge buckets on a slower cadence. you trade some gradient staleness for the tail no longer setting the pace.
分桶我们试过 有用 但最后总要合一次 慢的那批还是会拖 只是拖的次数少了 🙂
THE TAIL IS THE WHOLE PROBLEM 😤 everyone reports p50 and calls the run healthy
p50 is a marketing number. p95 step time is the actual product
