Theo Mendes

A short observation on inference pricing that I think is underappreciated: per-token pricing makes the customer bear the model's verbosity. If two providers serve the same quality at the same per-token rate, but one produces answers 40% longer, the second is meaningfully cheaper and no price sheet will tell you that. The comparable unit is cost per completed task, not cost per token, and almost nobody publishes it because it depends on your workload. Worth measuring yourself on a representative sample before committing. The ranking often inverts.

Something that keeps catching teams out when they compare inference providers: the cheap per-token price and the latency you need are the same lever pointed in opposite directions. Throughput comes from large batches. Large batches come from waiting for requests to arrive. If you hold a strict p99 you cannot wait, so batches stay small, the GPU sits partly idle, and your effective cost per token climbs well above the quoted one. The published price assumes a queue you do not have. The number actually worth asking for is cost per token at your latency target and your traffic shape. Nobody publishes it, because it differs for every customer, which is precisely why the headline price is such a weak signal.

final apex thought for a while: what the calibration work really buys you is the ability to build on top of the model without wrapping every call in your own uncertainty estimation. if the model reliably tells you when it's unsure, you can route those cases to a human or a fallback automatically. an overconfident model forces you to build a whole confidence-estimation layer yourself. calibration isn't just nice, it's load-bearing infrastructure for anything that composes the model into a larger system.

the affine adversarial-eval thing keeps making me think about how much of the ai leaderboard economy is basically measuring overfitting-to-known-tests. a static benchmark is a fixed target; the moment it's published, optimizing for it and optimizing for the underlying capability diverge, and after enough time the leaderboard measures the former. adversarial, population-based evals resist this because the target moves. noisier, harder to game, more honest. the field would be healthier with more of them and fewer static leaderboards.

circling back to bitmind after thinking about the arms-race problem more: the right way to evaluate a synthetic-media detector isn't its accuracy on today's generators, it's its accuracy on generators released after its last training update. that's the only test that measures generalization vs memorization. if bitmind published a rolling 'accuracy on novel generators' metric they'd instantly be the most credible detector in the space, because that's the number that actually predicts real-world performance.

the more i use ditto the more i think persistent memory is going to be table stakes for agent products within a year, and everyone faking it with retrieval hacks will have to rebuild. there's a real architectural difference between 'i stuffed your history into the context window' and 'i have a structured, queryable memory that persists and gets pruned intelligently'. ditto's on the right side of that line and it shows in the cross-session coherence.

one more chutes note because it's become my default: the underrated feature is that scaling to zero when idle means my experiment costs converge to actual usage instead of 'i forgot to shut down the box' bills. the cold-start tax is real but for the messy exploratory phase pay-for-what-you-use matters more than shaving 6 seconds off a first request. different tools for different phases.

spent the day watching ridges open PRs against a real codebase and the failure-mode analysis is more interesting than the successes. when it fails it's usually because it misunderstood an implicit convention nobody documented, which is exactly the tribal knowledge that makes onboarding humans slow too. in a weird way a coding agent is a great detector of your codebase's undocumented assumptions.

numinous doing probabilistic forecasting with a focus on calibration rather than point accuracy is exactly the right framing. a forecaster that says '70%' and is right 70% of the time is more useful than one that's confidently 'yes' and right 80% of the time, because you can size decisions off calibrated probabilities. if numinous optimizes for calibration and not vanity accuracy, that's a subtle but important difference from the pack.

coming back to apex after a few weeks and the web-grounding has noticeably improved, fewer confidently-wrong answers on recent events, better at saying 'i'm not sure' when sources conflict. that last part is underrated; a model that knows the edge of its knowledge is worth more than one that's slightly smarter but always certain. the macrocosmos team is clearly iterating.