What Is Quantization? The Cost of Shrinking a Model

What is quantization? Quantization is a model-compression technique that represents an AI model's weights with fewer bits — for example 8-bit or 4-bit instead of 16-bit — thereby shrinking the model's memory footprint and speeding up inference. In short, it leaves the model the same but stores the numbers inside it more cheaply; in return a small quality loss appears.

The most concrete problem with large language models is their size: billions of parameters, run directly, demand expensive, large GPUs. Quantization exists precisely to break through this wall — it makes a model runnable on small, cheap, accessible hardware. This guide covers, with the rigor of a consultant, what quantization is, why it is needed, how it works, the difference between levels like INT8 and INT4, how much memory saving it provides, where quality loss appears, the difference between PTQ and QAT, what methods like GPTQ/AWQ/GGUF do, when it is not appropriate, and how the quality of quantization is measured.

Definition

Quantization

A model-compression technique that represents an AI model's weights, and sometimes its activations, with fewer bits — for example 8-bit (INT8) or 4-bit (INT4) instead of 16-bit — thereby shrinking the model's memory footprint and speeding up inference. Because it rounds numbers onto a coarser grid, it produces a measurable quality loss; but INT8 is usually nearly lossless and INT4 trades a small amount of quality for a large memory saving.

Also known as: Quantization, model quantization, INT8/INT4 quantization, model compression

The shortest answer to what quantization is: representing a model's numeric values with fewer bits, that is, at a coarser resolution. A language model stores everything it has learned in billions of numbers (weights). These numbers are normally kept in 16-bit or 32-bit floating point form; each can express very fine differences over a wide range. Quantization deliberately reduces this fine resolution: it rounds each number onto a much smaller integer grid such as 8-bit (INT8) or 4-bit (INT4).

An analogy helps. Think of a high-resolution photo: it contains millions of color tones and is a large file. If you save the same photo with fewer colors, the file shrinks, but to the eye the difference is often unnoticeable; only if you look very carefully do you see some transitions become coarser. Quantization does this to the model: it shrinks the "color palette" of the numbers. The model takes up far less space, the output is nearly the same on most tasks, but a small quality loss can appear at sensitive points.

This distinction clarifies a critical point: quantization does not change the model's architecture, layer count, or what it learned; it only changes how it stores what it learned. The model is still the same model, carrying the same knowledge; it just represents this knowledge in a cheaper form. That is exactly why quantization is so attractive as a model-compression technique: it shrinks the model without touching the architecture and without requiring retraining. For the basics of how models work with numbers, the what is a tensor guide and, for what a model's parameters are, the parameters and hyperparameters guide are good starting points.

The most convincing answer to what quantization is comes from showing which problem it solves. The most fundamental constraint of large language models is their size. How many parameters a model has directly determines how much memory it needs; and this memory is usually expensive GPU memory (VRAM). If the model does not fit in memory, it does not run at all; even if it fits, the larger it is the slower and more expensive it runs.

Let us set up a concrete frame. In 16-bit (FP16) representation each parameter takes 2 bytes. By this simple arithmetic, a 7-billion-parameter model takes roughly 14 GB, a 13-billion one ~26 GB, and a 70-billion one ~140 GB of weight memory (these numbers are for weights only; activations and the KV cache add on top). These figures are not a benchmark but direct "parameter count × bytes" arithmetic. A 70-billion model does not fit on a single consumer GPU at FP16; but if quantized, the picture changes.

This is where quantization comes in. When you drop each parameter to 8-bit (1 byte) the model roughly halves, and to 4-bit (0.5 byte) it drops to about a quarter. So the 70-billion model falls to ~70 GB at INT8 and ~35 GB at INT4; it becomes able to fit on smaller, cheaper hardware. This memory saving is not just a matter of "fitting": a smaller model uses less memory bandwidth, which speeds up inference and lowers cost. We cover other ways to lower inference cost in LLM inference cost optimization.

As important as what quantization is, is how it works. The basic idea is to map a wide range of numbers (say all fractional values between -1 and +1) onto a finite number of integer steps. While a format like FP16 can express millions of different values, INT8 can hold only 256 different values (2 to the 8th) and INT4 only 16 different values (2 to the 4th). Quantization's job is to place the model's real numbers onto these limited steps with the least error.

This mapping is done with two small helper parameters: the scale and the zero-point. The scale determines how many steps the real number range is divided into; the zero-point adjusts where the grid is aligned. Quantizing a number means dividing it by the scale and rounding to the nearest integer; dequantizing means multiplying the integer by the scale. The small difference from rounding is the "quantization error" and is the source of quality loss. The goal is to keep this error small enough not to corrupt the model's output.

Let us see with an example why this rounding error is critical. Say a real weight has the value 0.732 and the quantization grid rounds it to the nearest step, for example 0.75. In a single weight this difference is negligible; but in a layer millions of weights are rounded at once, and these small errors combine to shift the layer's output a little. As the model gets deeper, each layer's small error carries to the next. A good quantization method, rather than leaving these errors random, tries to distribute them so they cancel out and to minimize them on the most critical weights. The aim is to preserve not the individual weights but the model's total behavior.

A critical detail is that quantization is done not with a single scale but in groups. Applying one scale to all weights is crude; because some weights are very large and some very small, and a single scale fits both poorly. So modern methods split weights into small groups (per channel, per block) and give each group its own scale. Also, some "outlier" weights are far larger than the rest; handling them specially (keeping them at higher precision) markedly preserves quality. What separates methods like GPTQ and AWQ is exactly how cleverly they make this "which weight gets how much precision" decision.

To fully grasp what quantization is, you must know in which formats models store numbers; because quantization is essentially a move from one number format to another. Models were traditionally trained in 32-bit floating point (FP32); each number takes 4 bytes and expresses very fine differences over a very wide range. But FP32 is wasteful for inference; so in practice one moves to 16-bit formats.

In the 16-bit world there are two important formats: FP16 and BF16 (bfloat16). Both take 2 bytes but strike a different balance: FP16 allocates more fraction bits (precision), while BF16 provides a wider exponent range (dynamic range). Because BF16 is more robust at representing very large and very small numbers, it has increasingly become the standard in modern training and inference. These two formats are taken as "reference quality" in practice; the baseline against which quantization is measured usually sits here.

Integer quantization (INT8, INT4) is the move from floating point to integers. While floating-point formats keep a number as "exponent + fraction," integer formats work on a fixed grid; this is exactly why scale and zero-point are needed — they turn floating-point flexibility into a fixed grid. In recent years another intermediate format has risen: FP8 (8-bit floating point). FP8 uses the same number of bits as INT8 but, because it preserves floating-point flexibility, can produce less quality loss on some tasks; it is especially supported on newer-generation hardware. For the numeric foundations beneath these formats, the what is deep learning guide and the what is a neural network guide, which show how the model's layers compute, provide context.

Common number formats and their practical positions

Format	Bits / bytes	Standout property	Practical role
FP32	32-bit / 4 bytes	Highest precision	Classic training, wasteful for inference
FP16	16-bit / 2 bytes	High precision, narrow range	Common inference baseline
BF16	16-bit / 2 bytes	Wide dynamic range	Modern training/inference standard
FP8	8-bit / 1 byte	Floating-point flexibility	Efficient inference on new hardware
INT8 / INT4	8-4 bit	Fixed grid, cheapest	The main target of quantization

The practical conclusion from this table: when we say "quantized model" we usually mean a move from a floating-point baseline (FP16/BF16) to an integer target (INT8/INT4). Which baseline you start from also matters; there can be subtle differences between quantizing a BF16-based model and an FP16-based one. That is why, when evaluating quantization, you should note not only the target level but also the starting format.

Quantization is not one thing; it is a spectrum of levels. Each level strikes a different balance between bit width and quality. The table below places the quantization level next to its impact, clarifying "which level trades away what" — the most practical answer to what quantization is.

Quantization level × impact: the balance of memory, speed and quality (quality effects are illustrative and task-dependent)

Level	Bits	Memory (vs FP16)	Typical quality effect	Best use
FP16 / BF16	16-bit	Baseline (1x)	Reference, lossless	Most quality-sensitive production
INT8	8-bit	~1/2	Very small, often unnoticeable	Balanced default
INT4	4-bit	~1/4	Small but measurable	Memory/cost-sensitive work
INT3 / INT2	3-2 bit	~1/5 and below	Noticeable, degrades fast	Experimental, not advised in most production

Three points to keep in mind while reading the table. First, "lossless" is a relative word: even FP16 is a quantization compared to 32-bit (FP32), but in practice it is taken as the reference. Second, memory ratios are arithmetic (bit-width ratio), but real memory use deviates a little due to activations, the KV cache, and scale parameters. Third and most important, quality effects are task-dependent and the qualifiers here are illustrative; you can only know the real effect for your own task by measuring.

The most common practical pair is INT8 and INT4. These two levels — the int8 int4 axis — cover nearly all enterprise quantization decisions: one leans on preserving quality, the other on minimizing memory. Intermediate levels between them (such as 5-bit or 6-bit GGUF variants) offer fine-tuning between these two ends; but the decision usually still comes down to the balance between int8 int4. Levels below INT4 (INT3, INT2) are interesting for research, but are not advised in most production scenarios because they corrupt quality too much. This curve between bit width and quality is not linear: dropping to INT8 is nearly free, while every additional bit cut costs progressively more.

Quantization's most concrete promise is memory saving and speed. Understanding these two separately is the best way to tie what quantization is to practice; because the two do not always come in the same proportion.

The memory side is the most predictable. Halving the bit width roughly halves the weight memory; this is direct arithmetic. Moving from FP16 to INT8 gives about 50% memory saving, and to INT4 about 75% memory saving. In practice this opens three doors: fitting the model on a smaller GPU, running a larger model on the same GPU, or running the same model with a larger context window (more KV cache). We cover how the context window consumes memory in what is a context window and the role of the KV cache in what is a KV cache.

The speed side is subtler. Quantization speeds things up in two ways: less data is moved (memory bandwidth gain) and, on some hardware, low-precision operations run faster (compute gain). Because most of inference is bound to reading data from memory (memory-bound), reducing memory traffic usually brings a visible speedup. But this gain depends on hardware: if your GPU lacks kernels optimized for that bit width, the model fits but the expected speed does not come. We detail what determines latency in LLM latency.

Quantization is not free; its cost is quality loss. But this loss is not spread evenly; understanding where and when it appears is the most critical part of what quantization is. The general principle: the loss grows as the level drops and becomes noticeable as the task gets harder.

On simple tasks the quality loss is often unnoticeable. On short summarization, classification, and simple Q&A, INT8 is practically lossless and even INT4 usually stays acceptable. The difficulty grows as the task gets finer and longer. Long chain reasoning, multi-step math, code generation, and sensitive numeric tasks are the areas most sensitive to quantization; because in these tasks a small error spreads to the next steps and grows. We cover how reasoning proceeds step by step in what is chain-of-thought.

A second face of quality loss is the "good on average, bad at the extremes" behavior. While a quantized model answers most questions at nearly original quality, it can err more often in rare and hard cases. This is missed when looking at average metrics; because rare errors lower the average very little but can matter in real use. A third effect is language and domain sensitivity: some quantized models preserve quality in English but degrade more in languages less represented in training data, such as Turkish. That is why in a Turkish-heavy application you must measure quantization with a Turkish test set.

Typical sensitivity of quality loss by task (illustrative, not exact values)

Task type	INT8 sensitivity	INT4 sensitivity
Classification / short summary	Very low	Low
General chat / knowledge	Very low	Low-medium
Code generation	Low	Medium
Long reasoning / math	Low-medium	Medium-high
Under-represented language (e.g. Turkish nuance)	Low	Medium-high

The practical lesson: quality loss is not a "general number" but a task-specific profile. A single sentence like "the INT4 model is 3% worse" is misleading; the right question is "what happens on my task, with my data, at my acceptability threshold." That is why a quantization decision is always a measurement decision.

When we say quality loss, "a wrong answer" comes to mind at once; but quantization's effect on the model's behavior is often subtler and sneakier. Recognizing these invisible effects gives an experienced answer to what quantization is; because a quantized model's problem is not always "clearly wrong," sometimes it is "a bit different."

The first invisible effect is loss of consistency. A quantized model may give slightly more variable answers to the same question at different times; the rounding error can push the output one way or the other in the border cases where the model is undecided. The second effect is a shift in style and detail: even when the model gives the right answer, it may write more briefly, more superficially, or with less nuance. This is invisible in the average accuracy metric but felt in the user experience. The third effect is a weakening in instruction-following: on complex, multi-condition instructions the quantized model may skip some conditions; because the precision carrying the fine distinctions has decreased slightly.

The fourth and most critical effect is error accumulation in long outputs. A language model produces text token by token, and each token becomes the context for the next; the small deviations quantization brings can accumulate step by step in a long generation and lower quality toward the end. That is why quantization is nearly invisible in short answers but more noticeable in long reports, long code, or long reasoning chains. We cover how the model produces text step by step in what is a token.

All of these effects scale with the level: while most are nearly absent at INT8, they become noticeable at INT4 and serious at lower levels. The good news is that these effects are predictable and measurable; the bad news is that if you do not measure them, they catch you in production, at the worst moment. That is why, when evaluating quantization, you should ask not only "is it correct" but also "how consistent, how complete, how robust in long outputs."

Quantization splits into two big families, and this distinction determines the most fundamental trade-off between quality and cost: post-training quantization (PTQ) and quantization-aware training (QAT).

PTQ, as the name suggests, quantizes an already-trained model without retraining. The model's weights are taken, suitable scales are computed with a calibration dataset, and the weights are rounded to low bit width. This process is fast and cheap — often minutes or hours — and requires no large training infrastructure. PTQ is the dominant way to quantize open-source models; popular methods like GPTQ and AWQ are PTQ techniques. We cover the details of PTQ in post-training quantization.

QAT rests on a different philosophy: it prepares the model for quantization during training or fine-tuning so that it gets used to low precision. Quantization error is simulated during training, so the model adjusts its weights to tolerate this error. The result is usually higher quality than PTQ, and the gap widens especially at very low bit widths (INT4 and below). But it has a cost: QAT requires training cost, data, and time. We detail QAT's logic in quantization-aware training.

When to prefer PTQ versus QAT

Criterion	PTQ (post-training)	QAT (training-aware)
Retraining	Not required	Required
Cost and time	Low (minutes-hours)	High (requires training)
Quality (low bit)	Good, struggles at extremes	Higher
Ease of application	High, ready tools	Low, needs expertise
Best use	Fast, most scenarios	Very quality-sensitive, aggressive quantization

The practical rule is clear: try PTQ first. In most scenarios PTQ gives sufficient quality at INT8 or INT4 and requires no training at all. Only if the quality loss falls below your threshold and low bit width is mandatory does moving to QAT make sense. For most organizations the right path is to start with the cheap and fast PTQ, and bear the cost of QAT only when there is a proven need.

There is a component that determines the quality of post-training quantization (PTQ) but is often overlooked: calibration data. In the technical answer to what quantization is, we mentioned scale and zero-point; these scales are computed over calibration data. The model is passed through a small set of sample texts, the ranges in which the weights and activations roam are observed, and the scales are set accordingly. So calibration data determines "which grid we place the numbers on."

This has an important consequence: if the calibration data does not represent real use, the scales are set wrong and quality loss grows needlessly. For example, if you calibrate a model only with English text and use it in a Turkish application, the range of Turkish activations may not be well represented; the result is a larger quality loss in Turkish. Likewise, calibrating a model to be used for code generation only with plain text may miss the code distribution. Good calibration data samples the content the model will actually see.

The amount of calibration data also matters but should not be exaggerated: usually a few hundred representative examples suffice; the aim is to capture the ranges statistically, not to retrain the model. The real issue here is not quantity but representativeness. For a Turkish-heavy application, putting Turkish examples in the calibration set is a detail as important as choosing the right embedding model; we also discuss a similar language-sensitivity issue in choosing a Turkish embedding model.

There is a common name confusion in the quantization world: GPTQ, AWQ, GGUF, bitsandbytes... Separating them is necessary to tie what quantization is to practical tool selection. Roughly two are methods (how to quantize) and two are more about format/library context; but the boundaries are intertwined.

GPTQ is a PTQ method that quantizes weights layer by layer while trying to compensate for the error at each step with the following weights; it is especially common for 4-bit inference on GPUs. AWQ (Activation-aware Weight Quantization) determines which weights are "important" for the output by looking at activations and preserves these important weights; it thus aims to keep quality better at low bit width. Both are in the PTQ family and share the same aim: minimizing quality loss at aggressive levels like 4-bit.

GGUF, by contrast, is not a method but a file format; it is used in the llama.cpp ecosystem to store quantized models and run them on CPU/GPU. GGUF offers not a single bit width but a range of quantization variants (for example 4-bit, 5-bit, 6-bit and different group settings); so for the same model you make a choice of "which GGUF quantization." Bitsandbytes is more of a library: it makes 8-bit and 4-bit quantization easy during training and inference and, in particular, forms the basis of fine-tuning with QLoRA. We cover practical ways to run a quantized model locally in what is Ollama.

Common quantization methods and formats (a rough orientation; the ecosystem changes fast)

Name	Type	Where it stands out
GPTQ	PTQ method	4-bit inference on GPU
AWQ	PTQ method	Quality by preserving important weights
GGUF (llama.cpp)	File format	CPU/laptop and flexible variants
bitsandbytes	Library	8/4-bit + QLoRA fine-tuning

Do not let the multitude of names scare you: the choice is usually determined by your hardware and tool. If you will serve on GPU, GPTQ/AWQ-based 4-bit is a common preference; if you will run on CPU or laptop, GGUF is the natural option; if you will fine-tune a quantized model, you follow the bitsandbytes/QLoRA path. What matters is not the name of the method but the quality and speed you measure on your target hardware.

Quantization is the most popular member of the model-compression family; but it is not the only member. There are several different ways to shrink and speed up a model, and distinguishing them is necessary to position quantization correctly. Most importantly, these techniques are not rivals but often complements.

Pruning sparsifies the model by removing unimportant or near-zero weights; while quantization stores each number more cheaply, pruning discards some numbers entirely. Knowledge distillation is a different idea: it teaches the behavior of a large "teacher" model to a small "student" model, producing a smaller model of similar quality. While quantization makes the existing model cheaper, distillation gives birth to a smaller model from the start. Low-rank adaptation (LoRA/QLoRA) is more an efficient adaptation technique than compression, but it is intertwined with quantization. We cover what LoRA and QLoRA are in what is LoRA and QLoRA, and the use of the three together in LoRA, QLoRA and distillation.

The nicest synergy among these techniques is seen in QLoRA: the model is first quantized to 4-bit (memory saving), then fine-tuned by adding small LoRA layers on top. This makes it possible to both quantize and fine-tune a huge model on a single GPU. Similarly, you can additionally quantize a small model produced by distillation, or combine two gains by quantizing a pruned model. For the approach of keeping the parameter count low from the start, the small language models and SLM fine-tuning guide also completes this picture.

Model-compression techniques: what they change

Technique	What it changes	Relation to quantization
Quantization	Bit width of numbers	The core technique
Pruning	Removes unimportant weights	Can be used together
Distillation	Transfers knowledge to a small model	Complementary
LoRA / QLoRA	Adapts with few parameters	QLoRA includes quantization

The conclusion: think of quantization not as an isolated trick but as part of an efficiency toolbox. The right strategy is usually not "quantization only" but a smart combination of these techniques according to the task and constraints.

Quantization is a critical lever not only for running a ready model cheaply but also for adapting a model cheaply. The nicest example of this combination is QLoRA, one of the scenarios that most concretely shows the practical value of what quantization is. QLoRA first quantizes a model to 4-bit (so the huge model fits in a single GPU's memory), then fine-tunes it by adding small, trainable LoRA layers on top. So fine-tuning, which normally requires very expensive hardware, becomes possible on a single modest GPU.

Why is this so important? Because fine-tuning a large model the classic way requires keeping all the weights in memory at high precision; that means hardware most organizations and individuals cannot reach. QLoRA breaks this wall thanks to the memory saving quantization provides. During fine-tuning the main model stays frozen and quantized; only the small LoRA layers are trained. We cover the details of this approach in QLoRA and the basic LoRA logic in what is LoRA.

There is a subtlety to note here: fine-tuning a quantized model strikes a different balance than running a quantized model. During fine-tuning, low precision can affect training stability; that is why methods like QLoRA use special techniques that keep the sensitive parts of the computation at higher precision. Still, the basic idea is the same: quantization is a common ground that makes both inference and adaptation cheaper. We evaluate when fine-tuning is needed and its alternatives in what is fine-tuning and the order of adaptation techniques in LoRA, QLoRA and distillation.

In conclusion, thinking of quantization only as "shrinking the model" is incomplete; it is also a tool for "making the model adaptable." An organization with limited hardware can, thanks to quantization and QLoRA, both run a large open-source model and customize it with its own data. This shows why quantization is not just an optimization trick but a capability that democratizes access.

After understanding what quantization is, the most practical question comes: which level in my case? The answer is not a single number but a decision made along a few axes. Here are the decision criteria for the choice between int8 int4 and staying at FP16.

The first axis is quality tolerance. The more sensitive the task is to error, the higher the precision you must keep. On low-error-tolerance work like legal analysis, medical information, and financial calculation, think of INT8 as the ceiling and FP16 as the safe harbor. On error-tolerant work like general chat, recommendations, and draft generation, INT4 is usually more than enough. The second axis is the hardware constraint. If the model fits your target hardware at FP16, you may not need quantization at all; if it does not fit, prefer the highest level that fits (INT8 first, INT4 if that is not enough).

The third axis is scale and cost. In a very high-volume, cost-sensitive service, the memory and speed gain of INT4 reflects directly on the budget; here a small quality loss can be traded for a large cost saving. The fourth axis is language: in relatively under-represented languages like Turkish the risk of INT4 is higher, so approach INT8 more cautiously on Turkish-heavy work. To make this decision as a whole, you must also think about how you will serve the model; we cover the options in model serving options.

In practice the most common decision around quantization is "should I download an already-quantized model, or quantize the model myself." In the open-source ecosystem many popular models are shared pre-quantized by the community at various levels (INT8, INT4, different GGUF variants). This often turns what quantization is from an engineering task into a simple download decision — but this convenience has its own traps.

The advantage of a pre-quantized model is clear: it is fast and effortless; you do not repeat work someone has already done. The disadvantage: you often cannot fully know with what data that model was calibrated, or with which method and settings it was quantized. This is risky especially in under-represented languages like Turkish; a quantized model produced with English calibration may show more quality loss than expected in your Turkish application. Also, the source's reliability matters: quantized model files, like other model files, should be obtained from trusted sources.

The advantage of quantizing yourself is control: you choose the calibration data according to your own use, you control the method and level, and you validate the result with your own evaluation set. The disadvantage is extra effort and expertise. A practical middle path: build a prototype with a pre-quantized model for a fast start, but be sure to measure it with your own task and language before taking it to production. If quality is not acceptable, consider re-quantizing with your own calibration data. We cover practical tools to run quantized models locally in what is Ollama and the open-source model selection frame in what is an open-source LLM.

Quantization is a powerful tool, but not a cure-all. In some cases the gain is not worth the risk, or quantization is unnecessary. Knowing these limits is part of giving an honest answer to what quantization is.

First case: if the model already fits. If you have hardware that can comfortably run a small model (say a few billion parameters) at FP16, quantization gains you very little but you take a small quality risk for nothing. Here the "if it is not broken, do not touch it" principle applies. Second case: if the task is extremely quality-sensitive. If the cost of an error is very high (critical decision support, safety, sensitive numeric processing), even the small quality loss of quantization may be unacceptable; here sacrificing quality for memory saving is a wrong trade.

Third case: aggressive quantization on very small models. Quantization is relatively safe on large models, because redundancy absorbs the error; but on very small models each weight is more critical and aggressive levels like INT4 can disproportionately corrupt quality. Fourth case: if there is no hardware support. If your target hardware lacks kernels optimized for low bit width, even if memory saving comes the expected speed does not, and the dequantize overhead can slow things down. Before hosting a quantized model in an enterprise, we assess hardware sizing in on-premise LLM hardware sizing.

Because quantization is a trade-off, you can only know whether that trade is worth it by measuring. The question "is the quantized model good" is not a guess but an evaluation question. Measurement is done in two layers: general language quality and task-specific success.

For general language quality the most common rough indicator is perplexity: a number measuring how well the model predicts a text. Looking at how much perplexity rises after quantization gives a quick sign of quality loss; but perplexity alone is not enough, because it does not always reflect real task success. We cover what perplexity is and its limits in what is perplexity.

Task-specific measurement is far more important. You prepare a labeled evaluation set suited to your use case, ask the same questions to both the FP16 and quantized models, and compare the results. If you generate code, you look at the test pass rate; if you summarize, at human evaluation; if you classify, at accuracy. The key is to look not only at the average but also at the distribution: even if the quantized model looks good on average, if it systematically degrades in certain hard cases you need to catch it. We detail the general methods of model evaluation in what is LLM evaluation.

Measurement must be done not once but continuously. The model is updated, the quantization tool changes, the usage pattern evolves; so building an evaluation set and re-running it at every change secures the quality quantization brings over time. To monitor this behavior in production you need observability; we cover this in LLM monitoring and logging. The essence of evaluating quantization: you cannot manage a quality loss you do not measure.

Quantization's value comes largely from its relationship with hardware. The same quantized model behaves very differently on different hardware; so to fully grasp what quantization is you must also see the hardware side.

On GPUs the main benefit of quantization is fitting larger models on smaller cards and relieving memory bandwidth. Modern GPUs offer special kernels for INT8 and increasingly INT4 operations; if these kernels exist, both memory and speed gains come together. We cover the enterprise role of the GPU in what is a GPU (enterprise AI) and the basic concept in what is a GPU. On the CPU side quantization opens a whole different door: thanks to tools like GGUF/llama.cpp, quantized models can run on ordinary GPU-less servers and even laptops. This is a critical option in scenarios that want to avoid high GPU cost.

Edge devices are where quantization provides the most dramatic benefit. The limited memory and processor of a phone, an embedded device, or an offline terminal can never handle an unquantized model; but an aggressively quantized model can run on these devices. So local inference becomes possible without data ever leaving the device — this is valuable for both latency and privacy. We cover the whole of local and on-premise running in on-prem LLM setup.

The golden rule of the relationship between hardware and quantization: do not choose the quantization level independently of the hardware. Knowing which bit widths your target hardware efficiently supports is a precondition for choosing the right level. Even if a level looks great on paper, if your hardware does not run it efficiently its practical benefit stays limited. So the decision must always be made together as a "model + level + hardware" triple.

Quantization is not the only way to speed up and cheapen a model; it is the most fundamental member of a larger family of inference optimizations. To place what quantization is exactly right, you must see it alongside the other members of this family; because real production systems use most of these techniques together.

The first neighboring technique is generation-acceleration methods like speculative decoding: generation speeds up because a small, fast model's guesses are verified by the large model. This is a gain channel independent of quantization and can be used with it; combining the two creates a multiplier effect. We cover this technique in speculative decoding and the holistic optimization of the inference service in LLM inference service optimization. The second neighbor is memory-management optimizations: efficient management of the KV cache in particular is the main factor determining memory in long contexts, and it relieves memory together with quantization.

The third related approach concerns the model's architecture: architectures like Mixture of Experts reduce compute by running only a part of the model for each token. This provides efficiency on a different axis from quantization and can be combined with it. We cover the MoE architecture in Mixture of Experts. The fourth is service-level techniques like batching and streaming; these raise throughput without changing the model.

The lesson of this picture: think of quantization not as an isolated solution but as part of an optimization stack. Quantization lowers memory and model size; speculative decoding speeds up generation; efficient memory management enables long context; batching raises throughput. The right production system is a smart combination of these techniques by task. We detail the relationship of latency with these techniques in LLM latency. Quantization is usually the first and highest-return step in this stack, because it lowers both memory and cost at once — but it produces the highest value not alone, but as part of a whole.

Quantization's enterprise value becomes clearest in on-premise LLM scenarios. For organizations where data must not leave, carrying KVKK and data-sovereignty concerns, running the model on their own infrastructure is attractive; but the biggest obstacle is hardware cost. This is exactly where quantization is a critical lever: it markedly lowers the required GPU investment.

Consider a concrete frame. An organization wants to run a large open-source model in-house. While at FP16 the model might require multiple high-end GPUs, INT4 quantization can fit the same model on far fewer cards; this lowers both hardware cost and the energy and cooling load. This memory saving can move an organization from "an on-premise LLM is not economical" to "now it is possible." We cover the decision between on-premise and cloud/API in self-hosted LLM or API, and the whole of the infrastructure in on-premise AI infrastructure.

But in an enterprise context quantization is as much a risk decision as an engineering one. Determining on which tasks the quality loss is acceptable is work to be done together with business units. For example, while the small quality loss of INT4 may be no problem in an internal knowledge assistant, an assistant doing contract analysis may need INT8 or FP16. The right approach is not to impose a single quantization level on the whole organization but to choose the level by task. For evaluating open-source models in an enterprise, the what is an open-source LLM guide offers a good frame.

Finally, enterprise quantization is not a one-off job. When the model is updated you must re-quantize, re-measure quality, and review the level decision. Organizations that treat quantization not as a "set and forget" step but as a continuous part of the model lifecycle build a sustainable balance in both cost and quality. This holistic view turns quantization from a trick into an enterprise capability.

Quantization is as much a cost decision as a technical trick; and the enterprise answer to what quantization is often ends with a budget sentence. Quantization's return comes through three channels, and each can be measured concretely: hardware cost, inference cost, and accessibility.

The first channel is hardware cost. Running a model at INT4 instead of FP16 can lower the number and class of GPUs needed; this reflects directly in capital expenditure (purchase) or operating expenditure (cloud rental). Given that high-end GPUs are expensive and sometimes hard to find, this memory saving can sometimes move a project from "not possible" to "cost-effective." The second channel is cost per inference: a smaller, faster model handles more requests on the same hardware; this lowers the cost per request. In a high-volume service, multiplied by scale, this means a serious saving. We cover ways to manage inference cost holistically in LLM inference cost optimization and the general cost frame in LLM cost optimization.

But a balance must not be forgotten in the return calculation: the cost saving quantization provides must be evaluated together with the impact of the quality loss it brings on the business. If dropping to INT4 lowers a support assistant's correct-answer rate and this means more escalation to humans, the hardware saving on paper can be erased in the real world. The right calculation is done not only with "how much GPU saving" but together with "the cost of the quality loss to the business metric." We detail how to calculate the return of AI projects in how to calculate AI ROI; the same discipline applies to quantization.

The third and often most valuable channel is accessibility. Quantization makes otherwise unreachable models accessible: a team with a limited budget can run a large open-source model on modest hardware; an organization whose data cannot leave can host the model on its own infrastructure; a developer can try a powerful model on their own laptop. Even though this accessibility does not appear directly as money saved, it is a strategic value that increases the speed of innovation and independence. Quantization's real return is often the sum of these three channels.

One of the best ways to make what quantization is concrete is to follow how a single decision is made. In consulting practice a typical scenario goes like this: An organization wants to run an open-source model on its own server instead of a cloud API for data-privacy reasons; but the available GPU cannot handle the model at FP16. The question is clear: should we quantize and fit it, or buy more expensive hardware?

The first step is not a hasty level choice but setting a baseline. The model is run at FP16 in an environment temporarily having enough memory, and its quality is measured with an evaluation set made of the organization's real questions; this is the reference for comparison. Then INT8 quantization is tried and measured with the same set. In most scenarios the quality difference with INT8 is imperceptible and the model now fits comfortably on the available GPU; in that case the decision is easy: INT8 is enough, expensive hardware is unnecessary.

But suppose even INT8 does not fit and you must drop to INT4. Here you must be careful: the same evaluation set is run again with INT4 and the quality loss is examined by task. If the organization's use is weighted toward general knowledge and summarization, INT4's small quality loss may be acceptable. But if the use involves sensitive numeric analysis or long reasoning, the drop at INT4 may cross the threshold; in that case you must either upgrade the hardware or take a more costly path such as quantization-aware training (QAT). The decision is not a blind choice but a measurement-based balance.

The lesson of this narrative: a quantization decision is never an abstract "which is better" debate; it is always a concrete decision made in the context of "my model, my task, my hardware, and my quality threshold." To make this decision tailored to your organization and build a model-efficiency strategy, the model serving options guide offers a good start; for a holistic roadmap, talking to an expert is the fastest path.

Understanding what quantization is in theory is easy; the hard part is applying it correctly in production. Seen with an experienced eye, failed quantization attempts break with similar mistakes. The most common are:

• Quantizing without measuring: The most common and most expensive mistake. Quantizing a model and taking it to production without measuring the quality effect on your own task makes a hidden quality loss invisible.
• Choosing the wrong level: Forcing INT4 on every job or staying at FP16 everywhere. The right level varies by task; making a single level a blind default is wrong.
• Not accounting for hardware: Choosing a bit width the hardware does not support; memory saving comes but the expected speed does not.
• Looking only at the average: Evaluating quality loss only by an average metric; missing degradation in rare but critical cases.
• Skipping language sensitivity: Deciding with English tests and missing the larger quality loss in under-represented languages like Turkish.
• Underestimating activation quantization: While quantizing weights is relatively safe, aggressively quantizing activations too without foreseeing the degradation caused by outliers.
• Thinking quantization is the only solution: Insisting on quantizing a large model in a situation where distillation, pruning, or a small model would be more suitable.

The most practical way to avoid these mistakes is to proceed small and measured: set a baseline, drop one level, measure, continue if acceptable. This discipline turns quantization from a gamble into a predictable engineering step.

The following checklist is a practical guide to quantizing a model soundly from idea to production. If you can tick these steps in order while turning "what is quantization" into an implementation, you have built a solid foundation.

How to

Quantization implementation checklist

A step-by-step checklist to safely quantize a model and take it to production.

1. 1

   Clarify the goal and constraint

   Why are you quantizing: to fit in memory, lower cost, or gain speed? Define the target hardware and the acceptable quality threshold from the start.

2. 2

   Set a baseline

   Before quantizing, run the model at FP16 and measure quality on your own task; this is the reference for comparison.

3. 3

   Prepare an evaluation set

   Build a labeled question-answer or task set suited to your real use case.

4. 4

   Try INT8 first (PTQ)

   Quantize to INT8 with training-free PTQ and compare quality against the baseline; often there is no difference.

5. 5

   Drop to INT4 if needed

   If you need more memory saving, drop to INT4 (GPTQ/AWQ/GGUF) and measure quality again.

6. 6

   Measure speed on hardware

   Measure real latency and throughput on your target hardware; verify whether memory saving turns into the expected speed.

7. 7

   Examine the distribution and extremes

   Look not only at the average but also at the quality loss in hard and rare cases.

8. 8

   Monitor and re-evaluate

   Monitor quality in production; re-quantize when the model is updated and review the level.

Following this checklist turns quantization from a guess into an evidence-based decision. A small but measured gain is always more valuable than a large but unverified promise. To build a quantization and model-efficiency strategy tailored to your organization, you can start with AI consulting, review corporate training options for your teams' competency, and deepen all concepts in the learning center.

What does quantization mean?

Quantization means representing a model's weights, and sometimes its activations, with fewer bits. A model normally stores each number at high resolution such as 16-bit (FP16); quantization rounds these numbers onto a coarser grid like 8-bit (INT8) or 4-bit (INT4). This way the model takes up less space and runs faster. In short, quantization is a way to store numbers with less precision but more cheaply; it changes how the model represents numbers, not what the model does.

How much does quality drop with quantization?

It depends entirely on the level and the task. INT8 quantization is nearly lossless on most tasks; users usually do not notice a difference. INT4 shows a small but measurable quality loss; on simple tasks the difference is almost nil, while on long reasoning, code, or sensitive numeric tasks the drop can become noticeable. Below INT4 (INT3, INT2) quality degrades fast. The key point: quality loss is not a single number; it cannot be known without being measured on your own task and data.

Which quantization level should you prefer?

The general rule: for quality-sensitive, low-error-tolerance work prefer INT8, because it is nearly lossless. For memory- and cost-sensitive work where you must fit the model on a small GPU or an edge device, prefer INT4, because it gives the largest memory saving. For highly quality-sensitive production work, either stay at FP16 or consider quantization-aware training (QAT). In every case, the decision should not be made without comparing the two options on your own evaluation set.

What is the difference between PTQ and QAT?

PTQ (post-training quantization) quantizes an already-trained model without retraining, within minutes or hours; it is fast, cheap, and the most common path for open-source models. QAT (quantization-aware training) trains the model to get used to low precision during training/fine-tuning; it gives higher quality but requires training cost and time. Practical rule: try PTQ first; move to QAT if the quality loss is unacceptable.

What is the difference between quantization and other model-compression techniques?

Quantization lowers the bit width of numbers. Other model-compression techniques target different things: pruning removes unimportant weights; knowledge distillation teaches a small model the behavior of a large one; low-rank adaptation (LoRA/QLoRA) adapts the model with a small number of extra parameters. These are complementary, not rivals; for example QLoRA fine-tunes a 4-bit quantized model with LoRA. The right approach is often to use them together.

Do you need special hardware to run a quantized model?

Usually no, but hardware support determines the gain. INT8 and INT4 kernels are supported on modern GPUs and increasingly on CPUs; tools like llama.cpp/GGUF can run quantized models even on ordinary laptops. Still, the real speed gain depends on the hardware having kernels optimized for that bit width. In some cases the model fits in memory but the expected speedup does not appear; that is why you must measure on your target hardware.

In short, the answer to what quantization is: a model-compression technique that shrinks a model's memory footprint and speeds up inference by representing its weights with fewer bits. The main gain is memory saving and speed — moving from FP16 to INT8 roughly halves memory and INT4 cuts it to about a quarter. The cost is quality loss: INT8 is usually nearly lossless, INT4 brings a small but measurable drop, and more aggressive levels degrade quality fast.

The most important message: quantization is not a guess but a measurement. The right level — the choice between int8 int4 or staying at FP16 — varies by task, hardware, cost, and language; and can only be known by measuring against a baseline with your own evaluation set. Start fast with PTQ, move to QAT if needed; think of quantization together with other model-compression techniques like distillation, pruning, and LoRA. For the basic concepts you can see the what is an LLM, what is fine-tuning, and what is deep learning guides; for a model-efficiency and quantization roadmap tailored to your organization you can get in touch with us and subscribe to our newsletter to follow the developments.