把训练集从本地挪到 hippius 上了 🎉 S3 接口直接能用 一行代码都没改 🙌 想问下有没有人测过冷数据取回 我这批一周才动一次 怕真要跑的时候卡在拉数据上 😅

把训练集从本地挪到 hippius 上了 🎉 S3 接口直接能用 一行代码都没改 🙌 想问下有没有人测过冷数据取回 我这批一周才动一次 怕真要跑的时候卡在拉数据上 😅
Depends what you mean by cold. In my testing the first read after a long idle stretch is slower but it is a single-digit second penalty, not a tier change like glacier. The thing that actually cost me time was listing large prefixes, not fetching the objects.
我这边跑过一轮。单个大文件没什么问题,麻烦的是几十万个小文件的目录,光遍历就要好几分钟。建议你先打包成分片再上传,取回速度差好几倍。
I would add that retrieval latency is probably the wrong number to optimise for a weekly access pattern. What bites you is the moment a run needs the whole set at once, and then it is the aggregate throughput ceiling you hit, not per-object latency. What I do is keep a manifest with sizes and a fixed shard order, so training can start on shard zero while the rest is still landing. That turns a cold start into a warm-up and makes the storage layer's latency mostly irrelevant to wall clock.
分片这个思路太顶了 🚀 我今晚就重新打包传一遍 谢谢两位 🙏🙏