What a token actually costs
"Inference is cheap" and "AI is burning money" are both true — at different layers of the same stack. This is the anatomy of inference cost, in the order the bills arrive.
The cost stack
| Component | What drives it | How it moves |
|---|---|---|
| Compute per token | Model size, sparsity (MoE routing), quantization | Falling: better silicon + better math |
| Context length | Attention cost grows with prompt size; long context multiplies everything | Your biggest self-inflicted lever |
| Utilization | Batch efficiency; idle GPUs bill at full rate | Why APIs win at low volume |
| Overhead | Retries, guardrail passes, orchestration calls | Often 20–50% of real spend, invisible in price lists |
The build-vs-buy line
API pricing is the retail cost of compute at high utilization. Self-hosting is wholesale compute plus your own utilization risk. The break-even is a volume line, and it moves: every API price cut raises the volume at which self-hosting wins. Teams that modeled self-hosting economics in 2024 should re-run the numbers quarterly — the answer flips.
The levers that actually move the bill
1. Right-size the model. Most production calls don't need the frontier tier; a smaller model at 10–30× lower unit cost clears the quality bar for classification, extraction, and routing work.
2. Cut the context. Retrieval discipline — sending the model what it needs, not the document — dominates every other optimization for long-doc workloads.
3. Cache and batch. Prompt caching and off-peak batch pricing convert bursty workloads into scheduled ones at a discount.
The engineer's takeaway
Model the full stack — context, retries, orchestration — not the price-list $/token. Then re-run the model quarterly; this curve moves faster than any other line in your architecture budget.
Independent technical analysis — not investment advice or engineering counsel.