GPT-4 launched in March 2023 at $60 per million output tokens. Today, Gemini 3.1 Flash costs $0.40 per million output. That is a 99.3% price reduction in three years. BenchLM.ai’s pricing index shows the average cost across frontier-class models has fallen 94.5% since April 2023. Most developers have absorbed this as a vague “things got cheaper” update. Very few have updated their architecture to match.
The RAG Pipeline You Built in 2024 Might Be Dead Weight
Retrieval-augmented generation existed, in large part, because stuffing data into context was expensive. At 2023 prices, loading a 500K-token corpus into every request was a real cost problem. At $0.10–$0.40 per million input tokens — the current price of Gemini Flash and DeepSeek — it costs fractions of a cent.
Research from wavect.io’s 2026 architecture analysis is direct: for datasets under approximately one million tokens, structured long-context prompts now outperform retrieval pipelines on both quality and maintenance overhead. Teams are decommissioning RAG stacks they spent months building. If your corpus fits in a one-million-token context window, audit whether the retrieval layer is still earning its complexity.
RAG still wins for large, dynamic, multi-source corpora. But that is a different use case than most teams were actually solving for in 2023. A lot of RAG implementations were cost workarounds, not architectural choices.
Routing Is Still Useful. The Math Changed.
In 2023, the routing case was easy: frontier models cost 10–30x the cheap tier. The math justified the engineering overhead. In 2026, frontier is 3–15x flash-tier depending on provider — and that gap is closing.
RouteLLM’s published benchmarks show their system achieves 95% of frontier model quality while sending only 14–26% of calls to the strong model, producing 75–85% cost savings on routed traffic. That is a real win — if you have the eval infrastructure to verify quality impact.
The problem is that routing infrastructure is not free. It adds latency, compute, and engineering overhead. When the margin between models was 20x, those costs were trivial. When the margin is 3x and your task is quality-sensitive, the routing machinery can cost more than it saves. The updated decision criteria: only implement routing if the quality tolerance is well-defined and testable, the price margin exceeds 5x, and you have evals to measure quality impact on production traffic. If any of those three conditions is missing, start with the frontier model.
The Output Token Trap
Here is the part of the 94% collapse story that does not make it into the headlines: output tokens cost 3–8x more than input tokens across every major provider. And reasoning models make this significantly worse.
Models like GPT-5.5 Sol, Claude Opus 4.8 with extended thinking, and OpenAI’s o-series generate internal chain-of-thought tokens before producing their answer. Those thinking tokens are billed at output rates — and you never see them. BenchLM.ai’s token pricing guide documents the effect: a single complex query can generate 10,000 invisible reasoning tokens. On GPT-5.5 at $30 per million output tokens, those hidden tokens cost $0.30 while the visible answer costs $0.006. You paid 50 times more for the thinking than the response.
Teams that upgraded to reasoning models without auditing thinking token ratios are discovering this on their bills. The mitigation is unsexy but necessary: instrument your API responses to track thinking token counts separately from completion tokens, and set effort-level controls where the API supports them.
Inference Is No Longer Your Biggest AI Cost Line
In 2024, inference typically consumed 70–80% of a production AI product’s operating cost. In 2026, that share has dropped to 30–45%. The rest has moved to vector database and search infrastructure, observability tooling, eval execution, and human review for agent products.
The practical implication: spending another sprint optimizing your model routing to save 15% on inference has lower ROI than it did two years ago. The evaluation pipeline is now the highest-leverage investment. If you do not have automated evals running on production traffic, you are flying blind on quality while paying for a cost problem that is largely already solved.
The Features You Killed in 2023 Are Worth Revisiting
According to The LLM Pricing Collapse of 2026, analyzing every code commit now costs roughly $1.50 per month for a small team. Per-user content personalization at one million events runs around $2,000 per month. AI-native search at 10 million queries: $5,000, down from $300,000. Deep agent loops with 6–10 tool calls dropped from $0.40 per task to $0.08.
Every product team has a list of features that were killed because the inference bill was too high. That list deserves a second look. The economics have changed by an order of magnitude in some cases, and items killed for cost reasons are not the same as items killed for product reasons.
The Moat Moved
When GPT-4 launched, access to the model was the moat. In 2026, everyone has capable models at commodity prices. The competitive advantage now lives in evaluation infrastructure, quality measurement systems, and the discipline to know when cheap intelligence is good enough and when it is not. That is a harder problem than buying API access. It is also a more durable one.
The constraint is no longer whether you can afford the API calls. It is whether you have the infrastructure to use cheap intelligence effectively. Check AI Pricing Guru’s comparison of 73 models to see where your stack sits today — then figure out what you are going to build with the budget you just freed up.
