There's a familiar arc to AI features in 2026. A team ships something genuinely useful, adoption climbs, and then a finance review lands: the inference bill is growing faster than revenue, and p95 latency has crept past the point where users start clicking away. The feature works. The economics don't. And because every design decision was made with the biggest model and the longest prompt, the fix looks like a rewrite.
It doesn't have to be. Here's how we think about the economics of an LLM feature from the first sprint.
Know your unit economics before you scale
Every AI feature has a cost per unit of value — per ticket resolved, per document processed, per conversation — and you should be able to state it. Instrument tokens in and out, model calls per task, cache hit rate and latency per step from day one, and tag every call with the feature and customer it served. Without that, "AI spend" is a single scary line on the cloud bill; with it, you know exactly which workflow is expensive and whether it's earning its keep.
Route by difficulty, not by default
The most expensive habit in AI engineering is sending every request to the frontier model. Most production traffic is routine: classification, extraction, reformatting, short answers over retrieved text. Small and mid-sized models — including fast-tier and open-weight options — handle these at a fraction of the cost and latency, often with equal accuracy on a bounded task. The pattern that works: a cheap model (or a rule) classifies the request, easy cases go to a small model, hard or high-stakes cases escalate to the frontier model, and your evaluation suite proves the routing didn't cost quality. Fine-tuning a small model on your own traffic often pushes even more work down the ladder.
Cache aggressively at every layer
Prompt caching on the providers' side makes long, repeated system prompts and shared context nearly free on subsequent calls — structure prompts so the stable part comes first and the variable part last. Above that, semantic caching returns stored answers for questions that are near-duplicates of ones you've already answered; in support and internal-knowledge use cases, hit rates of 20–40% are common. And below it, cache the expensive non-model steps too: retrieval results, embeddings, parsed documents.
Prompt hygiene is cost hygiene
Prompts accrete. Every incident adds a paragraph of instructions; every edge case adds an example; six months later the system prompt is 4,000 tokens and nobody knows which parts still matter. Audit prompts the way you'd audit dependencies: measure what each section contributes with your eval suite, cut what doesn't, and move stable reference material into retrieval so it's fetched only when relevant. Constrain output too — structured output schemas and explicit length limits stop the model from padding your bill with prose nobody reads.
Design for latency the user feels
Users don't experience average latency; they experience the slow tail and the blank screen. Stream tokens so something appears in under a second. Run independent steps in parallel instead of in a chain. Kick off retrieval speculatively while the user is still typing. Set per-step timeouts with graceful fallbacks so one slow provider call doesn't stall the whole task. And for multi-step agents, cap the step count — an agent that "thinks" for forty calls is a cost incident and a latency incident at once.
Batch what nobody's waiting for
A surprising share of AI workload isn't interactive: nightly document classification, enrichment, summarisation of yesterday's tickets, evaluation runs. Providers price asynchronous batch processing at a steep discount, and you can point it at smaller models with generous timeouts. Separating the interactive path from the batch path is often the single biggest line-item win.
The short version
Profitable AI features are engineered, not lucky: measure cost per unit of value from day one, route by difficulty, cache at every layer, keep prompts lean and outputs constrained, design for tail latency, and batch the work nobody is waiting for. Do that and scale becomes the goal again instead of the threat.
If your AI feature is winning users and losing money, let's look at the bill together.