# LLM Cost Optimization: Token Economics, Prompt Caching and Semantic Cache

> Source: https://sukruyusufkaya.com/en/blog/llm-maliyet-optimizasyonu-token-caching
> Updated: 2026-08-05T08:08:34.183Z
> Type: blog
> Category: yapay-zeka
**TLDR:** The strongest levers to control LLM cost in production: token economics, prompt caching, semantic cache and model routing, illustrated with 2026 pricing moves.

**TL;DR —** Over the past year I've watched the same scene play out at every company I've consulted for: the demo dazzles everyone, and then the first real invoice arrives and the room goes quiet. LLM cost, when not architected deliberately, compounds silently — because even though the per-token price looks tiny, the scale multiplier is merciless. In this piece I'll walk you through how token economics actually work, why output tokens cost so much more than input tokens, how prompt caching and semantic caching can cut your bill in half, how model routing lets you pair "cheap model + expensive model" strategically, and how batching, prompt compression, controlling agent loops, and cost observability all fit together in practice. I'll ground everything in the real pricing moves from late July 2026 — OpenAI slashing GPT-5.6 "Luna" by 80%, Anthropic's Claude Opus 5, DeepSeek V4-Flash, and Google's Gemini 3.6 Flash — to show that model choice and caching strategy are, today, the single biggest cost lever you have. At the end, you'll find a discussion of the self-hosting-versus-API trade-off in the context of Turkey's KVKK data protection law, plus a worked cost example from a Turkish enterprise.

## Why I care about this topic so much

When you put an LLM into production, the first conversation is always about accuracy and latency. But here's the real trap: the hard part shows up three months later. Usage grows, prompts get bigger, agents start calling more tools, and suddenly the monthly bill has tripled — without anyone doing anything you'd call "wrong." I've seen this happen with more than one client: at an e-commerce company, the monthly cost of a customer-support assistant grew fivefold in six months, simply because the system prompt kept expanding and nobody ever turned on caching. This article exists so you don't get hit by that three-month-delayed shock.

Don't treat cost optimization as "a performance tweak you add later." I think of it as an architectural decision, and it's most effective at the moment you first design the system. You can bolt it on afterward, sure, but it's far more painful, because by then your prompt structure, your agent flows, and your data model are already built in a way that isn't cache-friendly.

## Understanding token economics: why input and output are priced differently

No cost-optimization strategy makes sense until you understand the fundamentals of token economics. Put simply: with almost no exceptions, LLM providers price output tokens noticeably higher than input tokens — you'll typically see a gap of five to ten times. The reason is technical: input tokens are "read" by the model once and can be processed in parallel; but output tokens are generated autoregressively, meaning each one depends on the one before it. Every new token requires the model to re-evaluate the entire preceding context. That's computationally far more expensive, and the price reflects it.

The practical consequence of this asymmetry is that if you want to cut a system's cost, the first question to ask is: "Where is the money in this system actually going — input or output?" In RAG (retrieval-augmented generation) systems, input usually dominates, because large context windows, documents, and conversation history get fed into the model repeatedly. In work like code generation or long-form report writing, output tends to dominate. The strategy differs in each case: in input-heavy systems, caching is king; in output-heavy systems, capping output length, cutting unnecessary "thinking" steps, and picking the right model matter most.

One more thing to keep in mind: pricing tables are usually quoted "per 1 million tokens," and because that number looks small, people underestimate the danger. In a system handling 50,000 user requests a day, where each request involves 2,000 input tokens plus 500 output tokens, you're talking about 3.75 billion tokens a month. At that scale, even a $1 difference per million tokens turns into thousands of dollars a month.

## Late July 2026: a price war, and why it works in our favor

At the end of the summer of 2026, the pricing moves I watched unfold among LLM providers genuinely excited me as someone who consults in this space — because now, instead of telling clients "just wait, prices will come down," I can talk to them with concrete numbers. On July 30, 2026, OpenAI cut the "Luna" model in the GPT-5.6 family by 80%, bringing it down to $0.20 input / $1.20 output per 1 million tokens. The family's more capable model, "Terra," got a 20% cut, landing at $2 input / $12 output per 1 million tokens. On July 24, 2026, Anthropic released Claude Opus 5, which delivers intelligence close to Fable 5 while cutting its price roughly in half, and adds an "effort dial" — meaning you can now tune how much "thinking effort" the model spends on a given request, and therefore its cost. On July 31, 2026, DeepSeek V4-Flash was announced at $0.14 input / $0.28 output per 1 million tokens, standing out as a price-performance pick. And on July 21, 2026, Google's Gemini 3.6 Flash arrived as a cheaper, more efficient "workhorse" model — one that, in particular, cuts unnecessary reasoning steps, producing the same quality of answer while burning fewer tokens.

Here's a table summarizing these four moves:

| Model / Provider | Announcement date | Input price (per 1M tokens) | Output price (per 1M tokens) | Standout feature |
|---|---|---|---|---|
| GPT-5.6 "Luna" (OpenAI) | July 30, 2026 | $0.20 (80% cut) | $1.20 | Cheap, built for high-volume workloads |
| GPT-5.6 "Terra" (OpenAI) | July 30, 2026 | $2.00 (20% cut) | $12.00 | More capable, upper-mid tier |
| Claude Opus 5 (Anthropic) | July 24, 2026 | Near-Fable-5 intelligence, ~half price | ~half price | "Effort dial" to tune cost vs. capability |
| DeepSeek V4-Flash | July 31, 2026 | $0.14 | $0.28 | Price-performance leader |
| Gemini 3.6 Flash (Google) | July 21, 2026 | More efficient, lower cost | More efficient, lower cost | Fewer wasted reasoning steps |

The most important lesson I take from this table is: **model choice is, today, the single biggest lever you have on cost.** Using Luna instead of Terra for the same job can cut your input cost by a factor of ten and your output cost by roughly the same order of magnitude. That's not a blanket "always pick the cheapest" recommendation, though — the real point is to match the model to the workload, and to make that matching dynamic rather than static. Which brings us to the next topic, model routing. But first, let's look at the technique that delivers the fastest, most foundational win: prompt caching.

## Prompt caching: caching stable prefixes

I think prompt caching is the single biggest pile of money most teams leave on the table without realizing it. The logic is simple: in many production systems, a large chunk of the prompt sent with every single request never changes — system instructions, brand-voice rules, tool definitions, few-shot examples, frequently referenced documents. Only the user's latest message changes. Both Anthropic and OpenAI let you cache this stable "prefix" server-side, so that on subsequent requests carrying the same prefix, that portion doesn't need to be reprocessed and is billed at a much lower cost.

There are two benefits: cost, and latency. Because the cached portion isn't recomputed, it's both cheaper and returns faster — which directly improves user experience. I tell my clients all the time: "Turning on prompt caching is one of the rare optimizations that requires almost no engineering effort on the code side but shows an immediate, visible impact."

Systems that benefit most from prompt caching include:
- Customer-support bots with a fixed, lengthy system prompt
- RAG systems that repeatedly reference the same document set (you can cache the document content itself)
- Multi-step agents, where the same tool definitions and instructions get resent at every step
- Coding assistants, where instead of re-sending an entire codebase's context on every request, you cache it

There are a few practical things to watch for. First, the cacheable content needs to sit at the **beginning** of the prompt — because the caching mechanism relies on prefix matching, so anything that changes (the user's question, the current date, session info) always needs to go at the end. Second, caches have a validity window; if unused for a certain period, they expire, so expected gains shrink in very low-traffic systems. Third, if you change the cached content frequently (say, updating your system prompt daily), your cache hit rate drops. That's why drawing a clear line in your prompt engineering between "the part that never changes" and "the part that changes" needs to become an architectural discipline.

At one client, a 4,000-token system prompt (product catalog summary, brand tone rules, safety constraints) was being sent at full cost on every request. Simply fixing that content in place, moving it to the front, and turning on caching cut input costs by more than 60% — without touching the output side at all. That's a textbook example of "low effort, high return" optimization.

## Semantic cache: not regenerating answers for semantically similar questions

Prompt caching speeds up requests that share an identical prefix. But what if the user's question isn't exactly the same, just similar in meaning? That's where semantic caching comes in. The idea: run every incoming query through an embedding model to turn it into a vector, then compare that vector against the vectors of previously asked and answered queries. If there's a match above a similarity threshold you've set, you return the previously generated answer (or a small variation of it) directly, without ever calling the LLM.

This is especially valuable in scenarios like: FAQ bots, product support assistants, internal knowledge-base query systems. In an HR chatbot, "how do I use my vacation days" and "how do I request my annual leave entitlement" are worded differently but mean almost the same thing — semantic cache can route both to the same answer, skipping the LLM call entirely.

But I need to flag a warning here, because I've seen teams misapply this technique: semantic cache is **not safe in every context.** If there are subtle but important differences in meaning between questions (for example, "how many days for a return" versus "how many days for an exchange"), an overly aggressive similarity threshold can return wrong answers to users — and that can cost you far more in lost trust than you saved in dollars. So I recommend three rules to clients: keep the threshold conservative and test it against real traffic; give cached answers a TTL (time-to-live), especially for time-sensitive information like pricing, stock, or policy; and for high-stakes topics (health, legal, financial advice), either avoid semantic cache entirely or use it with a very narrow threshold.

Technically, semantic cache is usually built from a combination of a vector database (Pinecone, Weaviate, pgvector, or Redis's vector extension, for example) and an embedding model — which is generally very cheap, practically negligible compared to an LLM call. The cost comparison is stark: a single embedding call typically costs a thousandth, or less, of what an LLM call costs. So even a cache hit rate as low as 30% translates into a serious drop in your system's total cost.

## Model routing and cascading: don't ask the most expensive model every question

This is, I think, the most misunderstood but highest-return strategy out there. Many teams pick a single "best" model and route all traffic to it — under the logic of "let's use the best one, no compromise on quality." But here's the reality: the vast majority of incoming requests don't need the most expensive model's full capability. A simple classification question, a format conversion, a short summary — using the full power of Claude Opus 5 for these is like driving a truck to pick up a loaf of bread.

Model routing (sometimes called cascading) works like this: evaluate every incoming request first with a cheap, fast model (or a simple classifier). If that model can confidently resolve the query, end it there. If the model shows uncertainty (low confidence score, inconsistent answers, signals resembling "I'm not sure"), escalate the request to a higher-tier, more expensive model. It's exactly like a call center where a tier-one support rep handles the call first, and only escalates to a specialist team when they can't resolve it.

The pricing moves of summer 2026 make this strategy even more compelling. For example, you could build a three-tier system: use Gemini 3.6 Flash or DeepSeek V4-Flash (priced at fractions of a dollar per 1 million tokens) for simple classification and short answers; use GPT-5.6 Luna ($0.20 / $1.20) for medium-complexity tasks; and reserve Claude Opus 5, with its effort dial turned up, for genuinely hard, multi-step reasoning tasks. In this three-tier setup, if 70-80% of requests get resolved at the first two layers (which, in real-world traffic, is roughly the range you tend to see), your total cost drops by a large multiple compared to using a single premium model for everything.

The "effort dial" feature in Claude Opus 5 is particularly interesting here, because it shifts the routing decision from the model-selection level down to a "how much effort" level within a single model. In other words, you can always use the best model, but tune how "deeply" it thinks per task to control cost. That's a nice development that simplifies routing architecture.

A practical tip when setting up routing: you don't have to train your own classifier. A simple heuristic (request length, keywords, conversation history) or even using the cheapest model itself as a pre-check ("can you answer this — yes or no?") works fine. Don't chase perfection here; a routing decision that's even 60% accurate gives you a far better cost profile than no routing at all.

## Batching and asynchronous processing

For workloads that don't need real-time response (reporting, bulk data processing, overnight analysis jobs), batching is an overlooked but effective cost tool. Both OpenAI and Anthropic offer batch APIs for requests that don't require immediate responses — these get processed at a noticeable discount versus standard pricing, since it lets the provider use its server capacity more flexibly, in exchange for response times measured in hours rather than minutes.

I always ask clients: "Does the result of this job need to reach the user right now, or is it fine within a few hours?" If the answer is the latter — say, weekly customer segmentation, bulk content tagging, summarizing historical records — that work should absolutely go into a batch queue. Obviously slowing down a real-time chat experience with batching makes no sense, but dozens of background processes typically run over synchronous, real-time APIs unnecessarily, simply because "it's easier to code that way."

Asynchronous processing works on a similar principle: if you design your system to fire off parallel requests in the background without keeping the user waiting, you shorten total processing time and can take advantage of certain provider discounts. Especially in multi-step agent workflows, parallelizing sub-tasks that don't depend on each other (instead of running them sequentially) reduces both latency and, in some cases, cost, because it avoids repeatedly re-sending context during unnecessary wait periods.

## Prompt compression and context trimming

As context windows grow, it's tempting to think "more information = better answer," but that's not always true, and it's certainly never free. I recommend three techniques here.

First, **select only what's genuinely necessary.** In RAG systems, every document chunk you retrieve during the retrieval step turns directly into cost. Keeping the number and length of retrieved chunks tight, and using a reranker that measures relevance well (a second filtering layer after the embedding search), means you never feed unnecessary tokens into the system in the first place.

Second, **summarize conversation history.** In a long-running chat session, resending the entire history with every new message compounds token cost exponentially. Instead, after a certain number of messages, turning the older history into a summary and sending only that summary plus the last few messages keeps both cost and latency under control. This is especially effective in long-running customer support conversations.

Third, **prompt compression tools.** Some open-source tools (like LLMLingua) or simply your own rules (trimming unnecessary pleasantries, repetitive instructions, overly verbose examples) can reduce token count by roughly 20-40% without losing the prompt's meaning. Be careful doing this: aggressive compression can cause the model to misinterpret instructions, so testing response quality against real traffic after every change is essential.

One last point: trimming can also be applied on the output side. Leaving the `max_tokens` parameter needlessly high can lead the model to generate unnecessarily long responses (and you paying for them). Especially when you expect structured output (JSON, short category labels), clearly capping output length is a simple but effective savings technique.

## Controlling agent loops: the source of silent cost explosions

I want to emphasize this one in particular, because nearly all of the biggest "surprise invoice" incidents I've seen over the past year trace back to here. An agent calls a tool across multiple steps to complete a task, evaluates the result, and calls a tool again if needed. This loop is powerful by nature, but left uncontrolled, it can blow up cost in ways you never anticipated.

At one client, we hit a case like this: a research agent, on encountering an ambiguous search result, kept reasoning "let me try again" and called the same tool over and over — sometimes more than 40 times — resending the entire prior context to the model each time (because caching hadn't been set up correctly). A single user request that would normally cost a few cents sometimes ran up to several dollars. The only reason we caught it was that the bill that month came in unusually high — there was no alert, no ceiling, nothing.

The concrete controls I recommend to prevent this are:
- **Maximum iteration count:** Put a hard, task-appropriate ceiling on every agent loop. Don't let "let me try once more" logic run forever.
- **Budget-based cutoff (token/dollar budget):** An agent session should stop automatically once it exceeds a predetermined token or dollar budget, with an alert to the user or operator.
- **Loop detection:** Detect patterns like calling the same tool with the same parameters repeatedly, and cut them off early.
- **Timeouts:** Set time limits both per step and for the total session; this protects both cost and user experience.
- **Human approval threshold:** For agent flows that exceed a certain cost or step count, require human approval before continuing.

I recommend designing these controls not as "a safety net you bolt on later" but as an integral part of the agent architecture. Whether you're choosing an agent framework or writing your own agent loop, these limits need to be in the code from day one.

## Cost observability: you can't optimize what you can't see

This is, perhaps, the most overlooked step of all. Many teams only learn about their LLM cost from a lump-sum bill at the end of the month — that's like driving a car without ever looking at the speedometer. In production systems, you need to track the cost of every request in real time.

In practice, this means logging every LLM call tagged with which user/customer, which feature, which model, how many input/output tokens, and the computed dollar cost. Feeding this data into a dashboard (Grafana, Datadog, or your own simple internal tool) should let you see daily, feature-level, and customer-level cost trends. I always ask clients: "If I asked you right now which feature drove the most cost last week, could you answer?" Most of the time, the answer is no — and that's the clearest sign of why optimization tends to move so slowly.

Per-request cost tracking also gives you: early detection of abnormal usage patterns (like an agent loop spiraling out of control); visibility into which customer segment or feature affects your profitability the most; and the ability to prove, with A/B testing, whether optimizations like caching or routing are actually working. If you can't answer "did this change actually reduce cost?" with data, you're not really optimizing — you're guessing.

Finally, I strongly recommend setting up budget alerts — both at the provider level (spend limits in the OpenAI or Anthropic console) and at your own application level (a notification when a given customer or feature exceeds its daily/monthly budget). This is the simplest, cheapest insurance policy against surprise bills.

## Build vs. buy: self-hosting versus API

This is one of the questions I get asked most often, and the answer is almost never one-directional. The advantage of using the big providers via API is clear: zero infrastructure burden, continuously updated access to the newest models, scaling without worrying about capacity, and — especially with the price cuts of summer 2026 — cost that's now competitive for most workloads. Models like DeepSeek V4-Flash, priced at around $0.14/$0.28, make it harder for many mid-sized companies to justify the fixed cost of building and running their own GPU infrastructure (hardware, electricity, DevOps headcount, model-update overhead).

But there are scenarios where self-hosting (running an open-weight model on your own servers or in a private cloud) still holds real power: very high, sustained, predictable-volume workloads (where fixed cost can fall below variable API cost); systems that need very low latency in real time; and, most critically, **data sovereignty and regulatory requirements.**

In the Turkish context, this last point matters especially. Under KVKK (the Personal Data Protection Law), particularly in healthcare, finance, government, and some e-commerce scenarios, transferring personal data abroad is subject to strict rules. If a customer support system sends users' identity information, health history, or financial data to an LLM as part of the prompt, exactly which country that data is processed in, and whether the provider's data-processing agreement provides the guarantees KVKK requires, becomes a serious legal review matter. At that point there are three paths: first, use a provider's enterprise agreement that carries EU/appropriate-region data-processing guarantees; second, self-host open-weight models (families like DeepSeek, Llama, Gemma) within Turkey or in a KVKK-compliant cloud region; third, mask/anonymize sensitive data before sending it to the model (PII redaction) and only pass the necessary, anonymized context to the LLM.

When I'm consulting on this, I use a simple framework: if data sensitivity is high and volume is also high, seriously evaluate self-hosting. If data sensitivity is low (say, general product-info questions), the API is always a faster and cheaper starting point. And if data sensitivity is high but volume is low, PII masking plus an enterprise API agreement is usually a much more sensible middle path than the operational burden of self-hosting.

## A Turkish enterprise example: comparing costs with numbers

To make this concrete, let's walk through a simplified example inspired by my consulting experience. Assume a mid-sized Turkish e-commerce company processes 2 million customer support queries a month. Each query sends a fixed, roughly 3,000-token system prompt (product catalog summary, brand tone, return/exchange policies) plus a 150-token user question; the answer averages 300 tokens.

Let's compare this scenario using the real prices announced in July 2026, alongside reasonable (scenario-specific, illustrative) savings assumptions for caching and semantic caching:

| Scenario | Model | Input/output price (per 1M tokens) | Monthly input tokens | Monthly output tokens | Approx. monthly cost |
|---|---|---|---|---|---|
| No cache, powerful model | Terra | $2.00 / $12.00 | 6.3 billion | 0.6 billion | ~$19,800 |
| No cache, cheap model | Luna | $0.20 / $1.20 | 6.3 billion | 0.6 billion | ~$1,980 |
| Prompt caching on (assumption: the fixed 3,000-token portion is no longer billed at full price on repeat calls, but at a much lower effective rate), Luna | Luna | $0.20 / $1.20 | ~0.9 billion effective | 0.6 billion | ~$900 |
| Semantic cache + routing (assumption: 35% of queries are served from previously answered similar questions, with no LLM call at all), Luna | Luna | $0.20 / $1.20 | ~0.6 billion effective | 0.4 billion | ~$600 |

The bottom two rows in this table are not real company data — they're an **explicitly illustrative** calculation I've built for this article; the goal isn't to claim a precise figure, but to show how cost drops in stages as the right techniques combine. The structure, though, is real: choosing the right model alone (Luna instead of Terra) cuts cost by roughly 10x; adding prompt caching pulls input cost down further; adding semantic cache and routing on top reveals a savings potential exceeding 95% versus the first scenario. This shows you should think of this not as "each optimization is a small individual win," but as "a system where the effects stack and compound."

The KVKK dimension shouldn't be forgotten here either: if these customer support queries include personal data like ID numbers, addresses, or payment information, which model or provider you choose isn't just a cost question — it's a legal compliance question too. That's why I recommend treating cost optimization and data protection requirements as a single, unified design decision when architecting the system, rather than two separate concerns.

## A practical roadmap

If you're wondering where to start among all these techniques, my recommendation is to proceed in this order: first, make your current cost observable at the per-request level — you can't optimize what you can't see. Then turn on prompt caching, the lowest-effort, highest-return move; it's usually implementable in under a week and pays off immediately. Next, bring model routing online; start with a simple classifier, don't wait for perfection. If you're using agents, put step/budget limits in place right away, not "I'll add it later" — this is the riskiest cost source. Pilot semantic cache in low-risk, high-repeat-volume areas (FAQs, general support), then expand based on results. And finally, only consider self-hosting once your volume and data sensitivity genuinely justify it — for most companies, today's API prices, especially after the July 2026 cuts, are cheaper and carry far less operational burden than running your own infrastructure.

The biggest mistake I see when working with an organization is applying just one of these techniques and declaring "we've optimized." The real gain comes from setting all of these up together, as a system that reinforces itself. Understanding token economics tells you where to look; caching and routing tell you how to bring that cost down; observability proves whether those changes are actually working. Without all three together, LLM cost is destined to spiral out of control — and I've seen this happen even to teams that are excellent at their jobs, without them ever noticing.