把训练摊到一堆家用机器上这套东西，我断断续续跑了两个月，有些想法想记一下。 最大的障碍从来不是带宽，是异构。同一批里有 M2 的 Mac，有五年前的显卡，还有随时会被主人关掉去打游戏的机器。同步训练的话整批速度等于最慢那台，实测利用率低到不好意思贴出来。改成异步以后吞吐确实上来了，但你收敛到的东西已经不完全是原来那个目标函数了——旧梯度被当成新梯度用，等效于给优化器加了一层你没设计过的动量。 所以我现在的判断是：这类方案适合微调和小规模实验，不适合当作大规模预训练的替代品。不是因为算力不够，而是你很难向别人说清楚你的训练过程到底做了什么。
the stale gradient thing is real and nobody warns you about it up front. i spent a week convinced my lr schedule was broken when it was just staleness quietly doing its own thing to every update
有个折中做法是按延迟给梯度打折，越旧的权重越低。实现本身不复杂，难点在于需要每台机器上报可信的时间戳，在开放网络里这一步就已经不好做了。
I want to push back on the framing a little. Every large training run already does things you cannot fully explain, including the ones on tidy homogeneous clusters. Data order, hardware nondeterminism and mixed precision all leave fingerprints that nobody audits. The honest version of your point is narrower and I think stronger: heterogeneous async training makes the unexplainable part large enough that you can no longer pretend it is noise. That is a good reason to prefer it for fine-tuning over pretraining, but it is a difference of degree rather than of kind.
这个反驳我接受，确实是程度问题。不过我认为程度在这里恰恰是关键：同构集群上的不确定性大致可以按噪声处理，异步异构带来的偏差是有方向的，它会系统性地偏向那些跑得快的节点看到的数据分布。那就不只是解释不清楚，而是结果本身有偏。
