On-Premise LLM Setup: A Hardware Sizing Guide

For an on-premise LLM setup, the hardware requirement is set by three variables: the size of the model you will run (parameter count), the quantization level you apply, and the number of concurrent users you serve at once. These three inputs directly shape the GPU memory (VRAM) you need, the GPU count, and the final server sizing decision.

This guide focuses on a narrow, technical slice of the on-prem LLM topic — hardware sizing. The end-to-end architecture, network, storage, security, and operations side is a separate comprehensive guide; here the goal is to let you answer "how many GPUs, how much VRAM, which server" with a rough but reliable estimate once you pick a model. For the basics, what is an LLM and what is a GPU are good starting points. All numeric examples here are illustrative; exact values vary by model, runtime, and hardware.

Definition

LLM hardware requirement

The calculation of the GPU memory (VRAM), GPU count, and server capacity needed to run a language model on local (on-premise) infrastructure, based on model size, quantization level, and concurrent user load. Base formula: required VRAM ≈ weight memory + KV cache + operating margin.

Also known as: LLM hardware sizing, on-prem LLM hardware, GPU sizing, VRAM math

Hardware sizing depends not on a single number but on three variables that collide with each other. Understanding them separately turns the requirement from a guess into a rough calculation. The table below summarizes each input's effect on hardware and how it enters the math; starting sizing here is the soundest approach.

Sizing input, effect on hardware, and calculation approach

Input	Effect on hardware	Calculation approach
Model size (parameters)	Base VRAM taken by the weights	parameter count × bytes per parameter
Quantization level	Lowers bytes per parameter	FP16≈2, INT8≈1, INT4≈0.5 bytes
Concurrent users	Raises KV cache and throughput load	users × context × per-layer cost
Context length	Grows the KV cache linearly	increase proportional to token count
Operating margin	Buffer for activation and fragmentation	about 15-25% of the total

The logic is clear: the first two rows set how much space the weights take, the next two set how much extra memory is needed at runtime, and the last sets a safe buffer. The sum of these five items is the core of the GPU memory math you are after.

The starting point of sizing is the weight memory. The model-size-to-VRAM relationship is linear: the memory needed for a model's weights is the parameter count times the bytes per parameter. In FP16 (16 bit) each parameter takes 2 bytes; so each billion parameters takes about 2 GB. Thus a 7 billion parameter model needs about 14 GB in FP16, and a 70 billion parameter model about 140 GB of weight memory (illustrative values).

Why does this base figure matter? Because model size is the largest and most predictable component of the VRAM need; everything else is added on top. A single modern data-center GPU typically carries 24-80 GB of VRAM. So a 7-8 billion parameter model fits one card comfortably, while a 70 billion parameter model will not fit one card without quantization. To understand how a model splits text into pieces, see what is a token; parameter count and token-processing capacity are different things.

Two caveats matter here. First, inference and training are entirely different budgets: in training, gradients and optimizer state are kept alongside the weights, so memory need is several times that of inference; the calculations in this guide are for running the model in production, that is, inference. Second, VRAM is not the only bottleneck: to load the model from disk into memory you need fast storage at least the size of the weights and comfortable system RAM. Reducing sizing to the GPU's VRAM alone leads to forgetting disk and RAM and stalling on the first load.

Quantization is the technique of representing model weights with fewer-bit numbers, lowering model size and VRAM need. The clear answer to "what does quantization give you" is: it reduces the bytes per parameter. The 2-byte FP16 representation drops to 1 byte in INT8 and about 0.5 bytes in INT4. That shrinks weight memory by up to 2-4x — the roughly 140 GB FP16 base of a 70 billion parameter model drops to about 35 GB in INT4.

The gain is not only memory. Smaller weights consume less memory bandwidth, so throughput usually rises too, and the model becomes runnable on cheaper cards. The cost is a small accuracy loss, but with modern quantization methods it is acceptable in most enterprise scenarios. We assess options for running open-weight models on your own infrastructure in what is an open-source LLM, and local runtimes in what is Ollama.

A practical distinction: not every quantization level suits every job. INT8 is the safe default for most production loads because it halves memory with almost no accuracy loss. INT4 gives the most aggressive saving but can noticeably affect quality on complex reasoning or sensitive tasks; so a quality test on your own task is essential before moving to INT4. In scenarios where accuracy is critical and memory is not a problem, FP16 is still the right choice. In short, quantization is the strongest lever in on-prem sizing: it often runs the same model on half the hardware, but its level must be chosen deliberately.

The weights are only half the equation. While the model runs, a KV cache holds the intermediate state each active request produces, and this memory grows in direct proportion to the concurrent user count. In a single-user setup the KV cache is small; but if you serve hundreds of concurrent users, this cache can grow to rival the weights in total VRAM need.

Here two concepts must be separated: capacity and throughput. Capacity is how many requests you can hold in memory at once; throughput is how many tokens per second you can produce. High concurrent-user load strains both. In practice, modern runtimes raise throughput by processing requests in batches, but that too demands more KV cache memory. So server sizing must be done not by "does the model fit" but by "how many users can I serve at acceptable latency at the target load." To balance the cost side, LLM cost optimization is a good complement.

When the load spreads across multiple GPUs a new variable enters: the interconnect between cards. When a model does not fit one card, the weights are split across several GPUs (tensor parallelism) and the cards exchange data at every step; if this communication is slow, adding cards does not give the speedup you expect. So in a multi-GPU server not only total VRAM but also the speed of the interconnect binding the GPUs (a high-speed bridge or a standard bus) directly determines throughput.

The second factor that grows the KV cache is context length (the context window). Context is the number of tokens the model processes at once, and the KV cache grows linearly with it. Doubling the context roughly doubles the per-user cache memory; multiplied by the concurrent user count, long context alone can push total VRAM need past the weights.

The practical consequence: a server sized for an 8k-token context hits a memory wall when you try to run the same model at 128k tokens. We cover why context length is a resource decision in what is a context window. The golden rule in sizing is to choose the shortest context you truly need and to plan long context separately, only for the scenarios that require it; a "maximum context just in case" decision quietly multiplies the hardware budget.

Now let us tie these variables into a single calculation. The steps below are the practical way to turn a model into a rough but defensible number for the hardware requirement; they give a fast pre-screen before precise planning.

How to

Roughly sizing on-premise LLM hardware

Steps to approximate the required VRAM, GPU count, and server capacity starting from the model and load.

1. 1

   Define the workload

   Which model, target concurrent user count, and typical context length — clarify these three inputs.

2. 2

   Compute the weight memory

   Multiply parameter count by bytes per parameter: 2 for FP16, 1 for INT8, about 0.5 for INT4.

3. 3

   Add the KV cache share

   Add the runtime memory share considering concurrent user count and context length.

4. 4

   Add the operating margin

   Leave a buffer of about 15-25% of the total for activation and memory fragmentation.

5. 5

   Divide by GPU count

   Divide total VRAM by your GPU's VRAM and round up; turn the result into a server together with power and cooling.

This GPU memory math is deliberately rough; its aim is not a precise estimate but the right size class. Example: a 70 billion parameter model quantized to INT4 may need ~35 GB for weights and, with KV cache for moderate concurrency plus a 20% operating margin, ~50-60 GB total; this can be met by a single 80 GB card or two 40 GB cards (illustrative). Validating the real values in a small load test with your chosen runtime is essential.

Most sizing errors come from the same few misconceptions. Seen with an experienced eye, the most common are:

• Counting only the weights: Picking a card by the model's FP16 size means forgetting the KV cache; at high concurrency and long context the system unexpectedly fails to fit in memory.
• Overestimating context: Choosing maximum context "just in case" needlessly multiplies hardware cost; the context truly needed is much shorter in most scenarios.
• Ignoring quantization: Sizing in FP16 and inflating the budget is common; yet suitable quantization often runs the same model on half the hardware.
• Confusing throughput with capacity: Assuming the system will be fast enough at target load just because the model fits in memory; latency and throughput must be measured separately.
• Forgetting power and cooling: Server sizing is not only VRAM; the GPUs' power draw and heat load will bottleneck the deployment on site if not planned with data-center capacity.

The common root of these mistakes is choosing hardware before the workload. The right order is the reverse: measure the model and the real load first, then compute, then choose the card and server last. For a wider view of the local-model ecosystem, open-source LLM comparison also helps.

What hardware do you need for an LLM?

The core hardware for an on-premise LLM is one or more data-center-class GPUs, enough system memory to feed them, and fast storage. The decisive component is the GPU's VRAM, because the model's weights and the runtime KV cache live there. There is no single recipe: a 7-8 billion parameter model can run on one modern GPU, while a 70 billion parameter model usually needs several GPUs even when quantized. The right approach is to define the model and load first, do the GPU memory math, and then size the server accordingly.

How many GPUs do you need?

GPU count is set by whether the model's total post-quantization memory need fits into a single GPU's VRAM. Rough math: divide the required total VRAM by your GPU's VRAM and round up; even when weights fit one card, high concurrent-user counts and long context grow the KV cache and can force an extra card. For example, a 70 billion parameter model quantized to INT4 needs about 35 GB for weights, but with KV cache and margin it may spread across two GPUs in practice. If your throughput target is high, multiple GPUs sharing the load can beat a single large card.

What does quantization give you?

Quantization represents model weights with fewer-bit numbers, lowering model size and VRAM need. FP16 needs 2 bytes per parameter, INT8 drops to 1 byte, INT4 to about 0.5 bytes; that cuts the base memory need by 2-4x. The gain is not only memory: smaller weights consume less memory bandwidth, so throughput usually rises too. The cost is a small accuracy loss, but with modern quantization methods it is acceptable in most enterprise scenarios. In short, quantization is the most practical way to run the same model on cheaper hardware.

How does context length affect hardware?

Context length (the context window) is the number of tokens the model processes at once, and it grows the KV cache linearly. Doubling the context roughly doubles the per-user cache memory; multiplied by the concurrent user count, this load can exceed even the weights in total VRAM need. So in sizing you must account not only for model size but also for the context length you target.

What happens if GPU memory runs out?

The model can be shrunk with quantization, context length or concurrent users can be capped, or the load can be split across multiple GPUs. When VRAM is insufficient, offloading part of it to system memory is possible but drops throughput sharply and is rarely fit for production. The sound fix is to get the GPU memory math right from the start and plan server sizing to the real load.

In short, in an on-premise setup the hardware requirement is the product of three variables: model size sets the base weight memory, quantization sets the bytes per parameter, and concurrent users and context length set the KV cache. You add an operating margin to the sum of these items and divide by your GPU's VRAM to reach the GPU count, and from there the server sizing. Quantization is the strongest lever in this equation; it cuts the model-size-driven VRAM need by 2-4x and makes hardware accessible.

If you want a concrete sizing plan and a setup roadmap for which model, at which quantization, on which hardware for your organization, we can start with an AI consulting session and a GPU memory and server sizing study specific to your workload. For the end-to-end architecture, review the comprehensive guide, and to weigh the data sovereignty side, see the on-premises AI vs cloud comparison.