LLM Cost Optimization 2026: Prompt Caching, Batching, and Smart Routing

TL;DR — The strangest paradox of 2026: token prices fell roughly 80% since 2025 (GPT-4-level performance was about $30 per million tokens in early 2023, and about $0.40 today), yet enterprise LLM API spend passed $8.4B in 2025 and keeps climbing. "Prices fall while bills explode." The reasons are clear: inference is ~85% of enterprise AI budgets; ~60% of projects overshoot cost estimates by 30-50%; output tokens cost 4-6x input. This post puts the three real bill-reducing levers — prompt caching, batching, and smart routing — into an "LLM FinOps" framework: measurement/observability, stable prefixes, semantic caching, model cascades, output-length control, and a savings table. I close with a Turkey-specific note on TL/FX exposure and the self-hosting trade-off for KVKK/BDDK.

The sentence I've heard most from the companies I consult over the past year is this: "Prices are supposedly falling, so why do we pay more every month?" The question is fair, and the answer sits in a place management rarely enjoys: the problem isn't the model, it's the usage. I wrote this to bring together the cost map I draw on the whiteboard in those meetings and the tactics that actually saved money when we applied them. My goal is to give you a concrete playbook that can shrink your bill by 60-80%.

Unit price and total spend are two different things. Unit price falls, but total spend = unit price × tokens used. And tokens used has grown far faster over the last two years. There are a few reasons.

First, we consume more tokens per task. Where a single question-and-answer once sufficed, today one user request is split by an agent into dozens of steps; at each step long system prompts, tool schemas, retrieved documents, and reasoning traces are sent to the model again and again. Second, reasoning models produce visible tokens for "thinking," and those tokens hit the bill too. Third, success creates its own demand: once the LLM works, we put it in more places, and usage compounds.

You can't optimize what you don't understand. In practice, cost accumulates from these items: re-reading long system prompts on every request, ballooning chat history, documents retrieved by RAG, tool schemas, logs, reasoning traces, and long outputs. Most of this is repeating, static content — and that's exactly why it can be optimized.

On frontier APIs, output tokens cost roughly 4-6x input tokens. Internalizing this completely reorders your optimization priorities. Sending a 10K-token prompt and getting a 200-token short answer is often cheaper than sending a 2K-token prompt and getting a 3K-token long answer. "Make the model talk less" is, by itself, one of the highest-return levers.

The most common waste I see in the field: needlessly long outputs. The model explains in bullet points unasked, dumps rationale, apologizes, repeats. Putting a clear output contract in your system prompt — "return only the requested JSON, add no explanation" — noticeably lowers the bill. Start treating output length as a cost variable.

Reasoning models deserve special attention. These models produce long "thinking" tokens before arriving at an answer, and those are billed at the output rate. Running an expensive reasoning model on simple tasks is an invisible cost source: the user sees a three-sentence answer, but thousands of reasoning tokens were burned behind it. Tune reasoning depth (low/medium/high) to the task, and for simple work don't use a reasoning model at all; most requests don't need it.

Prompt caching is the highest-return and least-used technique of 2026. The logic is simple: the long, unchanging start of a request (system prompt, tool schemas, fixed instructions) is cached on the provider side; when the next request starts with the same prefix, the model doesn't process that part from scratch, it reads it from cache. It's both cheaper and faster.

A PwC study in January 2026 was striking: across OpenAI, Anthropic, and Google models, over 500 agent sessions with 10K-token system prompts were examined. The result: prompt caching cut API costs by 41-80% and improved time-to-first-token by 13-31%. Both cheaper and faster. But there's a critical condition.

"

Caches only help when the prompt prefixes stay stable. The cache matches from the prefix; if even the first character of the prefix changes, the match breaks and the entire advantage is lost.

The practical consequences of this condition are important. You must make your prompt design cache-friendly: put everything unchanging (system instructions, tool definitions, few-shot examples, fixed policy text) at the start of the prompt, and everything variable (the user's current question, timestamp, session-specific variables) at the end. A common-mistake warning: putting a dynamic date ("Today is July 24, 2026") at the very start of the system prompt changes the prefix on every request and renders the cache completely useless. Move dynamic values to the end.

Caches also have a time-to-live (TTL), usually from a few minutes to an hour. If your traffic is sparse, the cache cools and hit rate drops. So caching helps most in production systems with a continuous, similar stream of requests. For few, scattered requests, the gain is limited. Put cache hit rate on your dashboard as a metric; it's the most honest indicator of whether your prompt design is actually cache-friendly.

Prompt caching prevents reprocessing the same prefix. Semantic caching goes one step further: it aims not to regenerate answers to semantically similar questions. When a user asks "what's your return policy?" you cache the answer; when someone else says "how do I send a product back?", even though the words differ, because it's semantically close you serve the cached answer. The model is never called; cost drops to zero.

Semantic caching works wonderfully for frequently repeated questions whose answers don't change over time: FAQs, product info, policy questions. It's implemented with an embedding model and a similarity threshold: embed the incoming question, and if a sufficiently similar question exists in the cache, serve its answer. Caution: if you set the threshold too loose, you risk serving a wrong answer; avoid caching person- or context-specific questions. Build an allowlist to exempt personalized or time-sensitive answers from semantic caching.

Sending every request to the strongest (and most expensive) model is a comfortable but wasteful habit. Real traffic is heterogeneous: the bulk of requests are simple and can be answered perfectly by a small, cheap model; only a small minority genuinely need a frontier model's reasoning. Smart routing is the art of sending each request to the right model for its difficulty.

There are two core patterns. First, routing: a lightweight classifier (rule-based or a small model) estimates the request's difficulty and sends it straight to the appropriate model. Second, cascade: try the cheap model first; if the answer's confidence score or a validation test is sufficient, stop there; if not, escalate to the expensive model. A cascade solves every request the cheap model can handle cheaply and only hands off the hard ones expensively.

An observation from the field: in customer support and internal retrieval (RAG) workloads, most requests are surprisingly simple. In an e-commerce support bot, the bulk of traffic was questions like "where's my order," "how do I return"; routing these to a small model and leaving only complex complaints to the big model cut the bill significantly with no loss of quality. The key is to continuously measure the router's decisions and track the rate of misroutes (requests the small model botched).

A common trap when setting up routing is that the router itself is expensive. If you call a large model to estimate difficulty, you load an extra cost onto every request while trying to save. Keep the router as cheap as possible: try rule-based signals first (prompt length, keywords, tool requirement), and only fall back to a small classifier model when those aren't enough. The router must be far cheaper than the cost it protects.

For work that doesn't need to be real-time, batch APIs usually offer a meaningful discount. Give jobs like overnight bulk summarization, labeling a large dataset, email classification, and retrospective analysis to batch endpoints instead of real-time ones. Any job where no user is waiting in front of a screen is a batch candidate.

When placing batching into your architecture, split jobs into two buckets: interactive (a user is waiting, low latency is essential, real-time API) and background (no one is waiting, latency-tolerant, batch API). Making this distinction early both lowers cost and lets you reserve your real-time capacity for genuinely latency-sensitive work. In addition, combining many small requests into a single call (request-level batching) reduces fixed overhead.

These levers don't exclude each other; they multiply. When you apply prompt caching + batching + smart routing together, a 60-80% reduction in the bill is realistic depending on your traffic mix. Why multiplicative? Because each lever targets a different source of waste: caching kills repeated prefixes, routing kills unnecessary strong-model use, and batching removes the real-time capacity premium. The table below summarizes the levers and their typical impact.

Lever	What it targets	Typical impact	Where it helps most
Prompt caching	Repeated prompt prefixes	API cost down 41-80%, TTFT up 13-31%	Stable-prefix, continuously flowing traffic
Semantic cache	Semantically repeated questions	~100% cost cut on cache hits	FAQs, static info, policy questions
Smart routing	Overuse of strong models	Solves simple requests 5-10x cheaper	Heterogeneous, mostly-simple traffic
Batching	Real-time capacity premium	Batch API discount	Latency-tolerant background jobs
Output-length control	Expensive output tokens	Directly lowers output cost	Anywhere producing long/loose answers

To make the numbers concrete, let's set up a simple, illustrative scenario (these are for example only; your own measurements will differ). Say you have 100K requests a day, each carrying a 10K-token system prompt + 500 tokens of user input + 800 tokens of output, all going to the frontier model. First, turn on prompt caching and stabilize the prefix: processing the 10K-token system prompt is largely served from cache and the input side gets markedly cheaper. Then, with smart routing, shift, say, 70% of requests to a small model; that 70% drops to a small fraction of the unit cost. Finally, move latency-tolerant background jobs to batch. Each step stacks on the previous one, and the total effect exceeds the sum of the individual effects. The critical point: don't trust any estimate until you build this scenario with your own telemetry; every workload's mix is different, and savings depend on that mix.

Individual tactics are nice, but sustainable savings require a discipline. I adapt the cloud world's FinOps movement to LLMs. A three-stage loop: visibility → optimization → governance.

1. Visibility and measurement

You can't optimize what you don't measure. The two most fundamental metrics: cost per request and cost per resolved task. The second is critical because a cheap model can answer a request cheaply and give a wrong answer, the user asks three more times, and total cost rises. "Cost per resolved task" makes you think in outcomes, not tokens.

Log every request with the model used, input/output token counts, cache hit, latency, user/feature tag, and business outcome. Wire this telemetry into a dashboard. If you can't see which feature, which team, which customer segment burns how much, you're flying blind. Allocating cost by feature and team (chargeback/showback) is the only thing that changes behavior. Building observability from the start is far cheaper than bolting it on later, because all your optimization decisions will rest on this data.

2. Optimization

Once visibility arrives, prioritize the levers. The order is usually: first cut output length and prompt bloat (free, fast), then stabilize prefixes for prompt caching, then set up smart routing, and finally add batching and semantic caching. Validate every change with an A/B test: did cost drop AND was the task success rate preserved? An optimization that lowers cost but breaks quality is a hidden cost increase.

3. Governance

To make savings permanent, you need guardrails: per-team/per-feature budget limits, anomalous-spend alerts, a requirement that a new feature produce a cost estimate before going to production. Remember that ~60% of projects overshoot cost estimates by 30-50%; the antidote is to make cost an acceptance criterion at the design stage. "How much will this feature burn per request?" should be a standard line in the product requirements doc.

• Do we measure cost per request and cost per resolved task?
• Do we log every request with model, tokens, cache hit, and business outcome?
• Do we allocate cost by feature/team (showback)?
• Do the system prompt and tool schemas sit at the start of the prompt as a stable prefix?
• Did we move dynamic values (date, session variables) to the end of the prompt?
• Is prompt caching on, and do we track its hit rate?
• Is there a semantic cache for repeated, static questions, with a safe threshold?
• Did we set up a router / cascade that routes simple requests to a small model?
• Do we measure the misroute rate?
• Did we move latency-tolerant work to a batch API?
• Do we cap output length via the system prompt and a max-token limit?
• Do we tune reasoning depth to the task?
• Are there per-team/per-feature budget limits and anomalous-spend alerts?
• Do new features produce a cost estimate before shipping?

Everything so far is universal; but if you operate in Turkey, there are two extra dimensions. First, FX exposure. Nearly all frontier LLM APIs bill in US dollars. If your revenue is in TL, even if the token price falls in dollar terms, your TL cost can rise with currency moves. So the "prices are falling" news can be partly eaten by FX for you. Practical advice: plan budgets in dollars, not TL, and hold the TL equivalent with a buffer margin; set spend alerts in dollars; and where possible, negotiate an annual commitment/discount with the provider to fix your unit cost. FX volatility makes cost optimization even more urgent in Turkey, because you don't want to lose to the exchange rate what you gained from efficiency.

The second dimension is self-hosting and data residency. KVKK's rules on transferring data abroad and, for banking, the BRSA's (BDDK) cloud/outsourcing regulations make it harder in some sectors to send sensitive data to an API abroad, or tie it to extra obligations. In that case, hosting an open-weight model on your own infrastructure or in a Turkey-based cloud comes onto the table.

But thinking of self-hosting as a cost solution is a common fallacy. Make the comparison honest: GPU rental or purchase, scaling and idle capacity, an MLOps team, model-update and security burden are fixed and high costs. Self-hosting usually beats the API on unit cost only at high and stable volume; at low or fluctuating volume, the API is almost always cheaper. Your decision criterion shouldn't be pure cost but the combination of three axes: compliance obligation (do KVKK/BDDK require keeping data inside?), volume (does unit cost cross the break-even?), and quality (is the open model good enough for your task?). For most organizations, the healthy middle ground is hybrid: an internally hosted open model for sensitive, high-volume workloads, and the frontier API for everything else. This split lets you manage both compliance and the bill at the same time.

There are a few classic mistakes made in the name of "cutting costs" that actually work in reverse; knowing them in advance saves you both money and time. I've gathered the ones I see most here.

• Shrinking the model without measuring quality. Moving all traffic to a small model just because it's cheap delights you on the first bill, but if task success drops, users retry the same job again and again; total cost and dissatisfaction both rise. Measure every downsizing together with task success.
• Turning on the cache but leaving the prefix unstable. Enabling prompt caching and then putting a value that changes every time (date, session ID, username) at the start of the system prompt is the same as not enabling caching at all. Hit rate stays near zero and you can't tell why you aren't saving.
• Leaving observability for last. The "make it work first, look at cost later" approach creates, three months on, a black box where no one knows where the bill comes from. Measurement must be set up on day one.
• Retry storms. Aggressive retry logic on timeouts or errors can silently double or triple tokens. Use exponential backoff and a hard cap.
• Using context as a trash can. Stuffing every document, every log, the whole chat history into the prompt "just in case" causes both context rot and a bloated bill. Manage context with curation.

The common denominator of these anti-patterns: they're all easy in the short term and expensive in the long term. Establishing cost discipline as a continuous habit rather than a one-off project eliminates most of these traps from the start.

Cost is never optimized alone; it always moves together with its two neighbors, latency and quality. Any optimization that ignores this triangle creates a hidden price somewhere. For example, compressing output lowers cost and latency together — a rare win-win. But shrinking the model lowers cost while putting quality at risk; a cascade lowers cost but adds a second call on hard requests, raising latency.

My practical advice is to set a priority order across these three axes for each workload. In a chat interface where the user waits live, latency is king; there, streaming and prompt caching's effect on time-to-first-token stand out. In a background batch job, latency barely matters; there you pick the cheapest path, batch and the smallest sufficient model. In a high-stakes decision where an error is costly, quality is king; there, shrinking the model to save is false economy. Choosing different balance points for different workloads in the same system is a sign of mature LLMOps practice.

Let me underline one point: in this triangle there's no single setting that does everything best at once. Good engineering is deciding consciously which axis to sacrifice and when. You can't make that decision without measurement; that's why observability is the foundation of everything.

As important as the engineering levers, and often neglected, is commercial negotiation. Once you reach a high and predictable volume, it's often possible to negotiate discounts, provisioned throughput, or custom pricing with providers in exchange for committed usage. Fixing unit cost by contract both improves budget predictability and, specifically in Turkey, partly softens the impact of currency swings. Also, working with more than one provider (multi-provider) gives you bargaining power and protects you against a single price hike or outage. Think about cost optimization not only on the code side but at the procurement table too; the best result comes from the two working together.

Finally, tie these commercial decisions to telemetry too: measure which provider stands out on unit cost and quality for which workload. That way you sit down at the negotiation table not with intuition but with your own data, and back your discount requests with concrete volume commitments. The party that speaks with data has the advantage in every negotiation.

If you want a concrete starting plan, spend the first month like this. Week one, just measure: log every request, surface cost per request and per task on a dashboard, and identify the top three burning features. Week two, collect the free wins: tighten output contracts, trim prompt bloat, move dynamic values to the end to stabilize prefixes, and turn on prompt caching; this alone brings double-digit savings for most teams. Week three, pilot smart routing: route the highest-volume, simplest request type to a small model and watch the task success rate. Week four, set up governance: budget limits, spend alerts, and a cost-estimate requirement for new features. Once you've built this loop, the paradox where prices fall but bills grow turns in your favor: even as usage grows, your cost stays under control, because you now see and manage where every token goes and why.