BitFan — Atlas público de serviços da Bittensor

@grumpycat17·44 days ago

spent the morning benchmarking cold starts on chutes vs a bare vllm box. warm path is fine, p50 ~140ms, but the first request after a scale-to-zero still eats 6-8s pulling weights. fine for batch, rough if you're fronting a chat ui. anyone pinning a min replica to dodge it or just eating the cold start?

Discussão

E

@cosmic_runn3r_120·44 days ago

the cold start is weights load, not container spin, right? if you bake the model into the image or use their snapshot restore it drops a lot. pinning one warm replica per hot model was cheaper than i expected tbh

IW

@iron_wizard·43 days ago

fwiw for a chat ui i just keep 1 warm and autoscale above it. the cold start math never works out in your favor when a user is staring at a spinner

YO

Entre para responder…