The 2026 Guide to Cutting LLM Costs: Caching, Routing and Batching

TL;DR — In 2026 there is no single magic button to cut your LLM bill; but there are three layers that genuinely work when you stack them: model level (quantization, pruning, distillation), system level (continuous batching, PagedAttention, speculative decoding) and application level (context compression, prompt caching). Prompt caching alone can cut cost by 45-80% and improve time to first token by 13-31%; continuous batching gives 2-3x throughput over static batching; smart model routing saves 40-70%. Stack the three in the right order and a 60-80% bill reduction is achievable. In this article I walk through layer-by-layer concrete advice, the discipline of measuring first, Turkey-specific FX and KVKK/BDDK risks, and the token efficiency of Turkish, all from the field.

Nearly every team that comes to me about cost optimization makes the same mistake: they try to optimize before measuring. They start with "our bill is high, let us switch to a cheaper model" and weeks later come back having broken quality without capturing savings. But cost optimization is an engineering discipline, not guesswork.

That is why, above all, I say: optimization without observability is shooting arrows in the dark. Every step you take without knowing which request burns how many tokens, which endpoint eats the lion's share of the bill, which user flow is the most expensive, is a gamble. Your first job should be to set up simple monitoring that logs input/output token counts per request, model distribution, and cost per endpoint. I call this FinOps for AI: the discipline of cloud cost management, adapted to token economics.

After measuring, optimization proceeds in three layers. The table below summarizes these layers and the typical gain of each; in the rest of the article I will unpack each row one by one.

Layer	Techniques	Domain	Typical gain
Model level	Quantization, pruning, distillation	The model itself (mostly self-hosted)	Lower memory/compute cost
System level	Continuous batching, PagedAttention, speculative decoding	Serving infrastructure	2-3x throughput, low latency
Application level	Context compression, prompt caching, routing	Prompt and flow design	45-80% (caching), 40-70% (routing)

This layer is where you play with the model itself; it usually applies if you run models on your own infrastructure (self-hosted). If you use an API this layer is largely the provider's job, but it affects your decision.

Quantization means representing the model's weights at lower precision (for example 8 or 4 bit instead of 16). The result: less memory, faster inference, lower hardware cost. There is usually an acceptable, sometimes imperceptible, drop in quality. For teams running their own models, quantization is one of the techniques with the fastest payback.

Pruning means discarding weights or neurons with low contribution in the model; the model shrinks and speeds up. Distillation transfers the knowledge of a large "teacher" model to a smaller "student" model. A small model distilled for a specific, narrow task often does the work of a giant general model far more cheaply. The most efficient approach I have seen in the field is training a small, specialized model via distillation for narrow, repeating tasks.

The common theme of these techniques is this: running the world's largest model for every task is killing a fly with a hammer. Honestly ask how much intelligence the task actually requires.

This is about the efficiency of the infrastructure serving inference. If you host your own model it is directly your responsibility; if you use an API, how good the provider is at these techniques determines the price passed on to you.

Continuous batching is perhaps the most effective technique of this layer. In classic (static) batching, you wait for all requests in a group to finish; the slowest request holds up the whole group and the GPU sits idle. In continuous batching, the slot of a finished request is instantly filled with a new one; the GPU never sits idle. The result is a 2-3x throughput increase versus static batching. That means getting two to three times the work out of the same hardware; it directly lowers unit cost.

PagedAttention treats the attention mechanism's memory management like virtual memory in operating systems. It reduces memory fragmentation and lets you host more concurrent requests on the same GPU. Combined with continuous batching, the throughput gain compounds.

Speculative decoding is a small, fast model "guessing" a few tokens ahead, then the large model verifying these guesses in bulk. When the guesses hold, the large model does not need to produce tokens one by one; latency drops. An elegant trick that gains speed without sacrificing quality.

The beauty of this layer is that it gains efficiency entirely on the infrastructure side, without touching your application logic. Even if you use an API, when choosing a provider the question "how well do they implement these techniques?" should be central to your price/performance decision.

This is the layer where you have the most control and where most teams get the fastest gains. There are two big weapons: prompt caching and model routing.

Prompt caching

In most of your applications a large part of the prompt is identical on every request: the system instruction, tool definitions, few-shot examples, fixed context. Prompt caching processes this repeating prefix once and stores the result; on subsequent requests it pulls the same prefix from cache instead of processing it from scratch. The gain is twofold:

• 45-80% drop in cost: cached tokens are billed far more cheaply.
• 13-31% improvement in time to first token (TTFT): since the model does not reprocess the repeating context, it delivers the first response faster.

Where this helps most in the field is agents and RAG flows working with long, fixed system prompts. My advice: design your prompt so that the unchanging part comes first, the changing part later. Because the cache feeds on the prefix being fixed; if you put a variable part at the very start, the cache is useless.

Model routing

This is my favorite technique, because the gain is as big as the logic. The idea is simple: sending every request to the most expensive model is waste. You classify requests by difficulty and route easy ones to a cheap model and hard ones to an expensive model.

With a concrete example: send routine, simple requests to a cheap model (for example a Haiku 4.5 class, roughly 1 dollar per million tokens); reserve work that truly requires reasoning for the expensive model (for example an Opus class, roughly 5 dollars per million tokens). A striking figure from the field: 60-80% of code-agent requests are routine and never need the expensive model. When you shift these to the cheap model, you get 40-70% savings with almost no sacrifice in quality.

"

The rule is simple: running the smartest model for every request is like calling a taxi to buy bread from the corner shop. Ask how much intelligence the task truly needs, then bill accordingly.

Do not overcomplicate routing. Do not try to build a giant classifier from the start; begin with a simple rule set or a lightweight classifier. Cheap signals like request length, keywords, task type work surprisingly well. Over time, you measure the quality cost of misroutings and improve the classifier.

Now for the good part. These three layers are not mutually exclusive; they stack. In a suitable workload, when model-level gains, system-level throughput and application-level caching + routing combine, a 60-80% bill reduction is a realistic target.

But a warning: this figure holds "if the workload is suitable". Not every scenario can be compressed this much. In a system that receives highly varied, non-repeating, always-different and genuinely hard requests, caching helps little and the share of easy requests in routing is low. Be honest: measure how much of your own workload is routine and repeating, and calibrate your savings expectation accordingly. Exaggerated promises create disappointment later.

Now let us get to the part you will never see in international blogs but that is critical if you do business in Turkey.

FX risk. This is the most overlooked line in the field. Major LLM APIs are billed in USD. If your revenue is in Turkish lira, your bill is at the mercy of the TRY/USD rate. Even if your model runs with the same efficiency, your cost in lira terms can rise significantly within months due to currency moves. My advice: track your token budget and unit economics in both USD and lira; leave an FX buffer in your pricing. If you plan your cost only in dollars, an FX shock upends your budget overnight.

Data residency, KVKK and BDDK. The self-hosted versus API decision is not just a cost matter in Turkey; it is a compliance matter. Under KVKK, transferring personal data abroad creates additional obligations. If you are in the finance sector, the BDDK's regulations on data localization and outsourcing can directly constrain where data may be processed. In that case, even if API looks cheaper on pure cost, compliance may require you to move to a self-hosted model. In other words, your cost decision is not independent of law and regulation; you must discuss the two at the same table.

Token budget and unit economics. Every feature has a token cost per transaction or per user. Embed this into your unit economics: how many tokens does a transaction cost you, and how much revenue/value does it produce in return? Teams that do not do this math cannot understand why the bill grows disproportionately as users increase.

This is a fact everyone building Turkish products should know but most notice late. Language models split text into pieces called "tokens", and tokenizers are predominantly optimized for English. The result: a text carrying the same meaning can burn more tokens in Turkish than in English.

The reason is Turkish's agglutinative structure and tokenizers splitting Turkish word roots and suffixes inefficiently. A single Turkish word like "evlerinizden" can be split into far more tokens than its English counterpart. In practice this means: a Turkish application can carry a higher token cost for the same work than its English equivalent. Your bill can swell, without your noticing, purely because of language.

What to do? First, measure: compare the token density of your Turkish flows with English; see the surprise in numbers. Then avoid unnecessary length in prompt design; build system prompts more concisely in Turkish. Prompt caching is extra valuable here, because the cache absorbs the token load of the repeating Turkish context. Context compression techniques can also yield more in Turkish than in English, because there are more tokens to trim.

A frequently skipped technique of the application layer is context compression. The idea is to shorten the context you send to the model without distorting the meaning. In RAG flows, sending not all of the retrieved documents but only the parts genuinely relevant to the question; summarizing and carrying long conversation history; weeding out unnecessary repetition.

The most common waste I see in the field is piling giant context onto the model "just in case". This both burns tokens and can lower quality by distracting attention. Trimming context intelligently often gives both a cheaper and a more accurate result. Especially in Turkish, because of token inefficiency, the return on context compression is higher.

I said "measure first" but it would be unfair to leave that abstract. The metrics you need to watch when setting up observability are well defined; a team that monitors them regularly keeps both the bill and quality under control. The dashboards I build in the field always include these lines:

Metric	What it tells	Why it matters
Input/output tokens per request	Which flow burns how much	Key to finding the most expensive flow
Total cost per endpoint	Where the bill concentrates	Focuses optimization in the right place
Model distribution	How much traffic on the expensive model	Reveals the routing opportunity
Cache hit rate	Is caching actually working	If low, the prompt structure is broken
Time to first token (TTFT)	How long the user waits	Experience and caching effect
Cost-per-request trend	Is it rising over time	Early warning of uncontrolled growth

Do not set up this table once and forget it; look weekly. The most insidious problem I see in the field is the bill climbing quietly over weeks, not overnight. A team monitoring the cost-per-request trend catches this climb in its first week; a team that does not is startled by the month-end bill.

I mentioned prompt caching, but we are actually talking about two different approaches, and confusing them causes disappointment.

Exact-prefix cache captures the fixed prefix of the prompt verbatim; when the same prefix comes again it pulls from cache. This is the classic prompt caching providers offer. It is reliable and predictable; but it only works if the prefix is genuinely identical. That is why "fixed-first" prompt design is critical.

Semantic cache is a different idea: it caches the answer given to semantically similar questions. When a user says "what is your return policy?" and "how do I send the product back?", since both carry the same intent, you can return the same answer from cache. This gives enormous savings in customer service where repeating questions are common. But a warning: if the semantic cache mismatches, the user gets an irrelevant answer. So keep the similarity threshold conservative and audit false hits regularly.

My advice: start with exact caching, because it is risk-free and fast. Add semantic caching only if question variety is narrow and repetition is high, with careful thresholds.

Most teams try to process every request instantly (synchronously), yet a significant portion of the work has no urgency. For jobs like nightly report generation, bulk document classification, background summarization, you do not need a real-time response. You can save significantly by processing such work in batches and asynchronously; many providers offer discounted pricing for non-urgent batch jobs.

My habit in the field: I split the workload with the question "is a user waiting?". If the user is waiting at the screen, synchronous and fast; if not, asynchronous and cheap. This simple split moves a notable slice of the bill onto the asynchronous, discounted track. Most teams never exploit this opportunity, because they set everything up as real-time from the start.

Retrieval-Augmented Generation (RAG) flows are often evaluated only on the generation side when cost is discussed. But RAG has two hidden cost lines. The first is the embedding cost spent to vectorize documents; in large, frequently updated knowledge bases this is a serious line. The second and more insidious is the retrieved context bloating the generation prompt.

The mistake I often see in the field is RAG fetching ten to fifteen document chunks for every question and piling them all onto the model. This burns tokens on both the embedding and generation sides. The solution is twofold: limit the number of chunks you fetch to what is genuinely needed (reranking helps here) and send only the most relevant chunks to the model. On the embedding side, do not re-vectorize unchanging documents over and over; cache the embedding results. With one client, by intelligently trimming the number of fetched chunks alone, we halved the RAG flow's token cost, and answer quality even improved because the model worked with less noise.

The most dangerous aspect of cost optimization is the risk of quietly sacrificing quality. If a request type you shifted to the cheap model is actually too hard for that model, the money you saved is wasted as the drop in quality causes business loss. That is why it is essential to set up a quality assurance alongside every optimization.

In practice I do this: I keep a regression test set comparing outputs before and after optimization. When I change the routing threshold or shrink the model, I verify on this set that quality has not dropped. Automated evaluation (like LLM-as-judge) gives a fast signal here; but on critical flows I also sample with a human eye. The rule is: you may be sacrificing quality you cannot measure. Before celebrating the saving, show with numbers that quality stayed constant.

Answering this frequently asked question in the Turkey context by looking at token price alone is misleading. The decision must be made together with total cost of ownership (TCO) and compliance requirements. I compare the two options roughly like this:

Criterion	API	Self-hosted
Upfront cost	Low, starts fast	High (hardware, setup)
Unit cost (high volume)	Rises with volume	Can drop at scale
Maintenance/operations	On the provider	On you (team, expertise)
KVKK/BDDK data residency	Limited control	Full control
FX risk	High (USD)	Low apart from hardware
Optimization flexibility	Limited	Full (quantization etc.)

My general advice: at low and medium volume, if there is no compliance barrier, start with API; you gain speed and flexibility. As volume grows and FX risk and KVKK/BDDK constraints weigh heavier, self-hosting's TCO starts to make sense. But do not underestimate self-hosting; GPU cost, operational burden and the need for expertise are real lines. Running a model is easy; keeping it efficient and reliable in production is hard.

To bring abstract percentages down to concrete ground, let me describe, in simplified form, a process we went through with an e-commerce client over recent months. The monthly LLM bill of an assistant they built for customer service was climbing at an uncomfortable pace along with user growth, and the team was in a panic saying "let us switch to the cheapest model".

The first thing we did was not optimize but measure. We collected a week of observability data and saw this: most of the bill came from a single endpoint, namely the very long product catalog and fixed system instruction appended to every message. In other words, a giant unchanging context was being processed from scratch on every conversation turn.

The steps proceeded like this. First we reorganized the prompt: the fixed catalog and instruction went to the front, the user's changing message to the end, and we turned on prompt caching. This step alone largely melted away the cost of the repeating context and visibly shortened time to first token. In the second step we set up routing: we separated incoming messages with a simple classifier; routine questions like "where is my package", "how do I return" went to the cheap model, while genuinely complex complaints and edge cases went to the expensive model. Seeing that a very large share of traffic was routine surprised the team. In the third step we limited the product information the assistant pulled to only the relevant items; instead of the whole catalog, only the relevant products went into each message.

The combined effect of the three steps cut the bill to roughly a third, and while doing so the quality metric barely moved; on the contrary, the model working with leaner context gave more accurate answers on some flows. The lesson I drew is clear: the solution the team assumed (shrinking the model) was never needed; the real gain was hidden in measuring and starting from the right layer.

A technical detail, but it makes a difference in production. When a model generates text, it keeps in memory the key-value representations it computed for previous tokens; this is called the KV cache. If this cache is not managed efficiently, memory fills quickly on long-context requests, your concurrent-request capacity drops and unit cost rises. This is exactly the problem PagedAttention, mentioned above, solves: by managing the KV cache without fragmentation, in a virtual-memory-like arrangement, it fits more work onto the same hardware.

For teams self-hosting, the practical takeaway is: context length determines not only your token bill but also how many users you can serve at once. Unnecessarily long context penalizes you twice, through both direct token cost and indirectly reduced concurrency. That is why context compression is not merely an application-layer technique but also a lever that directly affects infrastructure efficiency.

Before the closing, let me list the expensive mistakes I encounter over and over in the field:

• Optimizing without measuring. The most basic mistake; every step taken without knowing which flow is expensive is blind.
• Running the biggest model for every request. 60-80% of routine work can be handled by the cheap model; not setting up routing is burning money outright.
• Putting the variable at the start of the prompt. The classic mistake that renders caching useless; the fixed part must come first.
• Bloating context "just in case". It both burns tokens and lowers quality.
• Ignoring FX risk. Planning a USD bill with lira revenue blows up the budget in an FX shock.
• Saving without measuring quality. Quietly dropping quality costs far more than the saving.
• Thinking optimization is a one-off project. Token prices and workload change; you must keep going back and measuring.

So in what order should you put all this into practice? The low-risk, high-return sequence I recommend to clients in the field:

1. Measure first. Set up observability; see token, model distribution and cost per endpoint. Without this step, progress is a blind flight.
2. Find the biggest line. The bill usually concentrates in a few endpoints. Target the most expensive flow; do not waste time on small optimizations.
3. Turn on prompt caching. Usually the fastest and lowest-risk gain. Reorganize the prompt to be fixed-first.
4. Set up routing. Shift routine requests to the cheap model with a simple classifier. Adjust the threshold while monitoring the quality metric.
5. Compress context. In RAG, send only the relevant part; summarize history. In Turkish the return of this step is higher.
6. Descend to the system/model layer. If volume and compliance justify self-hosting, bring in continuous batching, PagedAttention and quantization.
7. Account for FX and compliance. Embed the TRY/USD buffer and KVKK/BDDK data residency constraints into the decision.

When you follow this order, you get the biggest gain with the least effort early, then deepen gradually. Cost optimization is not a one-off project but a habit that requires continuity; as token prices, models and your workload change, you need to regularly go back and re-measure. It is the teams that continuously measure and tune, not those who cut the bill once and forget, who win.