Why Inference Matters 12x More Than Training | YC Paper Club

This week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade. But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality. We brought together researchers and builders from Together AI, Ollama, Luminal, and SGLang & RadixArk to take us through that stack from end to end. We’ll see how