LLM Inference Serving Optimization: vLLM, Speculative Decoding, and KV Cache Quantization

TL;DR — If you self-host an LLM, most of your bill and latency come not from the model itself but from how you serve inference. vLLM's PagedAttention cuts memory waste by up to 90% and multiplies throughput, speculative decoding can lower per-token cost by 20-54%, and KV cache quantization shrinks memory manyfold. In this piece I explain each of these techniques — PagedAttention, continuous batching, speculative decoding, KV cache compression, quantization — from the field: when each is useful, and how to decide in Turkey's self-hosted scenarios (KVKK, cost).

Let me start with a misconception I see repeatedly in the field. A team decides to run an LLM on their own servers, picks a strong model, rents GPUs, and then is surprised by two things: the bill is higher than expected and latency is worse than expected. The first reflex is usually "let's get a bigger GPU" or "let's use a smaller model." But most of the time the real problem is neither the model nor the hardware; it's how you serve inference.

LLM inference works fundamentally differently from a classic web service. The model produces text token by token; each new token is computed by looking at all tokens produced so far. This creates two cost sources: memory (holding the key-value representations of past tokens) and compute (running the whole model for each token). A naive serving setup uses both of these terribly inefficiently — most of the memory is wasted, and the GPU often waits. Modern inference optimization is precisely about eliminating this waste.

In this piece I'll explain how you can achieve manifold throughput and cost improvements without buying hardware or shrinking the model, just by setting up the serving layer correctly. These aren't theoretical tricks; they're measured techniques running in production. And the best part: for most of them you don't have to wait for a new budget approval or hardware order; it's about using the resources you already have more intelligently.

Where it all begins is the KV cache. As the model produces text, it keeps the "key" and "value" representations of past tokens in a cache so it doesn't recompute them for each new token. Sounds reasonable, but there's a problem: this cache grows enormous as the produced text lengthens. In a long conversation or a long document summary, the KV cache memory can take even more space than the model's weights.

Naive serving setups manage this memory very poorly. They allocate memory upfront for the longest possible output for each request — reserving space for 2000 tokens "in case I produce 2000 tokens," but if the request ends at 200 tokens, the remaining 1800 tokens' space is wasted. Across thousands of concurrent requests this waste compounds, and your GPU memory fills with reservations that are never actually used. The result: you can process few requests at once, and your bill runs while your GPU sits idle.

This is why KV cache management is the heart of inference optimization. The more efficiently you use memory, the more requests you process at once on the same hardware, and the lower your per-token cost. Most modern techniques focus, directly or indirectly, on solving this memory problem.

What makes vLLM so effective in the inference world is an elegant idea called PagedAttention. This idea is actually borrowed from the virtual-memory and paging logic operating systems have used for decades.

In the classic approach you keep the KV cache as a single, contiguous memory block — which leads to both waste and fragmentation. PagedAttention instead splits the KV cache into small, fixed-size "pages" and allocates these pages on demand, as needed. Just like an operating system splitting memory into pages and allocating them when needed. So a request only holds as much memory as it actually uses; it doesn't reserve a huge block "just in case."

The practical effect is striking: PagedAttention can reduce memory waste by up to 90%, which makes many more concurrent requests possible on the same hardware — in some scenarios boosting throughput up to 24x. Moreover, the page logic makes sharing easy too: hundreds of requests using the same system prompt can share that prompt's KV cache pages, saving you from doing the same computation over and over. In enterprise scenarios where everyone uses the same long system prompt, this sharing alone provides serious savings.

"

Field practice: If you serve an LLM on your own infrastructure with a raw setup (e.g., loading the model directly and processing requests one by one), you're probably using a small fraction of your hardware. Moving to a PagedAttention-backed server like vLLM often means "5-10x more work on the same GPU." This is not a code change but an infrastructure change, and usually the highest-return first step.

The second big lever is how requests are batched. GPUs are amazing at doing many computations in parallel at once; but that parallelism is wasted when processing a single request. So batching requests and processing them together is the key to throughput.

Classic "static batching" works like this: you gather a few requests, start them all together, and wait until they all finish. The problem: requests finish at different lengths. If one request in the batch finishes at 50 tokens while another takes 1000, the short request's slot stays idle until the longest one finishes, even though it's done. So the fast finishers wait on the slow ones and the GPU is used inefficiently.

Continuous batching solves this problem. When a request finishes, it immediately takes a new request into the batch in its place — without waiting for anyone. So the GPU runs constantly full, with no idle slots. This makes a huge difference especially in real-world workloads where output lengths vary greatly. vLLM and similar modern inference servers do this natively; you don't have to set it up separately, but knowing why it matters helps you choose the right server.

Now to one of the most elegant techniques: speculative decoding. The idea here is genuinely clever and targets a bottleneck inherent in token generation.

The problem: the big model produces each token one by one and must run the entire model for each token — which is slow. Speculative decoding inserts a small, fast "draft model" in between. This small model quickly guesses a few tokens ahead; then the big model verifies these guessed tokens in a single parallel pass. If the guesses are correct (which, with a good draft model, they usually are), the big model has approved multiple tokens in one pass — meaning you produce multiple tokens at the cost of one.

The critical point: the output is mathematically identical to what the big model would produce alone. So you don't sacrifice quality; you only speed up. In practice, with optimal tuning, gains of 20-54% in per-token cost are reported. The size of the gain depends on how well the draft model predicts the big model; high prediction accuracy means a big gain, low accuracy a small one. So the draft-model choice and the "how many tokens ahead to guess" (the k value) tuning matter; the optimal k differs per workload and you must find it by measuring.

Quantization is representing model weights and/or the KV cache at lower precision (e.g., 8, 4, or even fewer bits instead of 16). This both shrinks memory and speeds up computation. For a long time it was approached cautiously with a "does it reduce quality?" worry, but in 2026 quantization techniques matured so much that in most scenarios the loss is nearly unmeasurable.

You can quantize two places separately. Weight quantization makes it easier to fit the model in memory and lets you run it on a smaller GPU. KV cache quantization is critical in long contexts: techniques like Google's TurboQuant can compress the KV cache to 3 bits and achieve a 6x memory reduction with no measured accuracy loss. This fundamentally changes the cost of long conversations or long document processing.

But quantization isn't a blind "always on" button. The more aggressively you quantize, the higher the accuracy risk; and this risk varies by model, task, and language. In a language like Turkish, aggressive quantization may cost slightly more quality than in English. So the golden rule: measure accuracy before and after quantization on your own task, in your own language. Don't quantize blindly trusting general benchmarks.

These techniques aren't mutually exclusive; they layer. A good production setup uses most of them together. You use memory efficiently with PagedAttention, keep the GPU full with continuous batching, speed up token generation with speculative decoding, and shrink the model and KV cache with quantization. Each layer adds gains on top of the others.

Technique	What it improves	Main benefit	Watch out
PagedAttention	Memory management	Up to 90% less waste, high throughput	Comes with server choice
Continuous batching	GPU utilization	No idle slots, high throughput	Critical with variable output
Speculative decoding	Generation speed	20-54% per-token gain	Draft-model accuracy decisive
Weight quantization	Model memory size	Smaller GPU, cheaper	Accuracy-precision balance
KV cache quantization	Context memory	Manifold savings in long context	Extra measurement in Turkish

But my warning: don't turn them all on blindly at once. Each technique has a complexity and risk cost. Start with the highest-return and lowest-risk one — which is usually moving to a PagedAttention-backed server and enabling continuous batching. Then measure your workload, find the bottleneck, and add the technique suited to that bottleneck.

To optimize inference correctly, you need to know that a single request actually goes through two different phases; because these two phases are entirely different in character and require different solutions. The first phase is "prefill": where the model reads the entire input prompt and prepares to produce the first token. This phase is compute-bound; it uses the GPU's full power because it processes all tokens of a long input in parallel. The second phase is "decode": where the model produces the answer token by token. This phase is memory-bandwidth-bound; it reads the model from memory over and over for each token but can't fully use the GPU's compute power.

Why does this distinction matter? Because your system's bottleneck changes depending on which phase your workload emphasizes. Long prompts and short answers (e.g., document summarization, RAG) are prefill-heavy; here your input-processing speed is critical. Short prompts and long answers (e.g., creative generation, long code) are decode-heavy; here your token-generation speed is critical. The same optimization doesn't give the same result on both workloads. For example, speculative decoding speeds up the decode phase but barely touches prefill; prefix caching, on the other hand, fundamentally changes prefill. You can't choose the right technique without profiling your own workload.

Most enterprise LLM usage reuses the same long system prompt across thousands of requests: "you are the assistant of this company, follow these rules, speak in this tone..." If this prompt is the same in every request, computing its KV cache from scratch each time is enormous waste. This is exactly what prefix caching solves: the common prefix is computed once, the result is cached, and all subsequent requests continue from where it left off without redoing that computation.

The effect is striking with long common prompts. If most of your prefill cost comes from the common prefix, prefix caching drops that cost to nearly zero and dramatically shortens time to first token. There's a similar opportunity in RAG scenarios: if the same document chunks repeat across many questions, their representations can be cached. Modern inference servers support this feature increasingly well; when choosing a server, always check prefix/prompt caching support because one of the biggest savings in enterprise workloads comes from here.

In multi-tenant scenarios, different customers' different prompts must fairly share the same GPU. The challenge here is that one tenant's long request must not starve the others. Continuous batching and a good scheduling policy provide this fairness; but under high load you may also need to set up prioritization and quota mechanisms. If you run an enterprise platform, designing this resource-sharing issue from the start comes far cheaper than firefighting later.

An advanced approach that matured in 2026 is separating the prefill and decode phases into different hardware pools (disaggregation). The logic: since prefill is compute-heavy and decode is memory-heavy, running both on the same GPU makes both suboptimal. If you separate them and run each phase on hardware suited to its need, overall throughput rises. This is an architecture that provides serious gains in large-scale services but also increases operational complexity; unnecessary for small setups, but it should be on your radar if you run a high-volume platform.

Another practical point on scaling: load is variable. LLM workloads usually rise and fall through the day; low in the morning, high at noon, near zero at night. Keeping a fixed number of GPUs on 24/7 burns money during idle hours. Auto-scaling (add GPUs when load rises, remove when it falls) significantly lowers cost; but since LLM servers have long startup times, you must build your scaling policy accounting for this latency. Keeping a small buffer capacity to absorb sudden load spikes is usually the right balance.

Fortunately you don't have to write most of these techniques from scratch; there's a mature open-source ecosystem. vLLM is one of the most widely used options offering PagedAttention and continuous batching natively. Different inference servers have different strengths; some are faster on certain hardware, some better support certain quantization formats, some offer certain features (speculative decoding, prefix caching) more maturely. When choosing, look at fit to your own hardware, model family, and workload.

My advice is to test with your own golden set rather than chasing trends. A server being fast on a general benchmark doesn't mean it'll be fast and accurate on your specific workload and Turkish content. Create a small representative load, measure a few candidate servers under that load — for speed, accuracy, and operational ease. This measurement takes a day or two but the cost of choosing the wrong tool lasts for months.

Let me gather the mistakes I see repeatedly in this area. First, staying with a raw setup: loading the model directly and processing requests one by one uses a small fraction of the hardware. Not moving to a mature server is ignoring the biggest gain on the table. Second, optimizing without measuring the bottleneck: prefill or decode, memory or compute — optimization done without knowing this often targets the wrong place. Third, turning on quantization blindly: aggressive quantization without measuring accuracy on your own task causes silent quality loss. Fourth, not measuring Turkish: a setup that works perfectly in English can weaken unexpectedly in Turkish. Fifth, optimizing once and forgetting: the workload changes, yesterday's tuning may not be optimal today.

The common root of these mistakes is the same: deciding without measuring. Inference optimization is a discipline that runs on metrics, not intuition. The more you measure, the more accurately you optimize.

In inference optimization, saying "it got faster" isn't enough; you must clearly measure what you sped up, because different workloads care about different metrics. There are three core metrics. First, time to first token: the time from the user sending the request to seeing the first word. In interactive applications like chat, this determines perceived speed. Second, inter-token latency: how fast each new token arrives after the first. This determines the fluency of long answers. Third, throughput: how many total tokens per second the system produces. This determines your cost-efficiency; the higher it is, the lower your per-token cost.

These metrics sometimes conflict. For example, if you batch aggressively, throughput rises but a single user's time to first token may lengthen. So optimization is striking a balance dependent on your workload: prioritize latency in interactive chat, throughput in batch processing. Instead of a blind "speed up everything" goal, determine which metric is critical for you and tune accordingly.

To make optimization decisions correctly, I recommend focusing on a single number: the real per-token cost. Compute it like this: divide the total amount you pay the GPU over a given period by the total tokens you produced in that period. This number is the final report card of all your optimization effort. Whatever you do — PagedAttention, continuous batching, or quantization — in the end look at whether this number dropped.

This perspective clarifies a lot. For example, if your GPU costs a fixed amount per hour and you can produce twice the tokens in the same hour, your per-token cost has halved — without changing the model, without paying a cent more. Conversely, if your GPU mostly sits idle (low utilization), your per-token cost becomes astronomical because you pay for every idle second too. This is why throughput optimization is directly a cost optimization; the higher the throughput, the more tokens you spread the fixed hardware cost across, lowering unit cost.

A caution: while optimizing per-token cost, don't lose sight of quality. Aggressive quantization lowers unit cost, but if it also lowers accuracy, you may actually be producing "cheap but useless" answers — which is the most expensive outcome. So track cost and accuracy together; the ideal point is the lowest unit cost at acceptable accuracy, not the lowest cost. Find this balance by measuring on your own task and revisit it over time.

These techniques have a special meaning for organizations in Turkey. Many Turkish organizations, due to KVKK and sector regulations (like BDDK in banking), prefer to run the model on their own infrastructure rather than send data to a foreign API. Self-hosting is reassuring for data privacy but has a cost: you now manage inference cost and performance. This is why the techniques above are "a nice detail" for those using external APIs but a direct bill-and-user-experience matter for those self-hosting.

Let's think concretely. GPUs are expensive and access in Turkey is sometimes hard. Getting as much work as possible out of the hardware you have isn't a luxury but a necessity. With PagedAttention and continuous batching you can serve manifold more users on the same GPU, with quantization you can fit onto a smaller, cheaper GPU, and with speculative decoding you can lower latency and raise user satisfaction. So these techniques are what make self-hosting economically sustainable.

There's also a hybrid approach: process sensitive data with a self-hosted model on your own infrastructure, and route non-sensitive, high-accuracy-requiring work to an external API. This is a practical strategy balancing KVKK, cost, and accuracy. When deciding which work goes where, the self-hosted side's inference efficiency directly affects this balance; the more efficient it is, the more work you can economically keep in-house.

Let me wrap up with a practical roadmap. If you self-host an LLM and cost/latency is straining you, do the following in order. First, move to a mature inference server supporting PagedAttention and continuous batching; this alone usually gives the biggest leap. Then measure your workload — is it time to first token, throughput, which metric strains you? If the bottleneck is memory, add quantization (especially KV cache); if the bottleneck is generation speed, try speculative decoding. After each step, re-measure accuracy on your own task, in your own language, so you don't silently lose quality for speed.

Let me add one more thing: most of these improvements come nearly "for free" when you choose the right tools. PagedAttention and continuous batching kick in without extra engineering burden once you pick a mature server; you just have to choose the right one. Speculative decoding and quantization take a bit more tuning but the return-to-effort ratio is high. So most of the techniques I described in this piece aren't a months-long R&D project; they're a few-week, measurement-driven improvement round. What's critical is doing it not by intuition but with your own workload's real profile. Measure, find the bottleneck, add the technique suited to that bottleneck, measure again — this loop lowers your inference cost manifold over time.

The biggest mistake is thinking optimization is a one-time project. Your workload changes, the model updates, your user count grows; yesterday's optimal tuning may not be optimal today. So monitor your metrics continuously and re-tune regularly. Inference optimization isn't something set and forgotten but a living part of your infrastructure. But once set up correctly, the work you extract from the same hardware multiplies and the economics of self-hosting change entirely — an option that looked expensive and hard becomes a sustainable, competitive strategy.