Capacity calculator
What can your GPUs actually serve?
Select the hardware you already have. This works out which models fit, at what precision, across how many GPUs, and how many concurrent sequences the memory left over will hold. Every number has an arithmetic derivation you can check.
No form, no email, no account. The results are useful whether or not you ever talk to us.
The window you intend to serve, not the model's maximum. Sizing for 128k when your requests are 4k wastes most of the cache.
Auto picks the highest precision that fits, working down from bf16.
fp8 KV halves cache memory and roughly doubles concurrency. It is not free, measure quality on your own evals before shipping it.
vLLM's --gpu-memory-utilization. The remainder holds the CUDA context, activations and CUDA graphs.
Models to check (7 selected)
4 GPUs · 319 GiB of device memory · 4 × H100 SXM 80GB
You could run
4 × H100 SXM 80GB
319 GiB total · sm90 (Hopper) · 3,350 GB/s
| Model | Weights | Placement | Concurrent at 8k | Show arithmetic |
|---|---|---|---|---|
| llama-3.1-8b8.03B | bf1615.0 GiB | 1 GPU4 replicas | ~56224 across the group | |
| llama-3.3-70b70.55B | bf16131.4 GiB | TP ×41 replica | ~62concurrent | |
| mistral-7b-v0.37.25B | bf1613.5 GiB | 1 GPU4 replicas | ~58232 across the group | |
| qwen2.5-32b32.76B | bf1661.0 GiB | TP ×22 replicas | ~4182 across the group | |
| qwen3-30b-a3b30.5B total · 3.3B active | bf1656.8 GiB | 1 GPU4 replicas | ~1976 across the group | |
| gpt-oss-120b116.8B total · 5.1B active | bf16217.6 GiB | TP ×41 replica | ~242concurrent | |
| sarvam-12B | bf163.7 GiB | 1 GPU4 replicas | ~77308 across the group |
Concurrency is the number of sequences whose KV cache fits simultaneously at the full context length. Requests shorter than 8k use proportionally less, so real fleets run more than this. One sequence costs 1.0 GiB for the first model listed.
What this cannot tell you
Memory feasibility is arithmetic, and this is the same arithmetic vLLM does at startup. Expect real vLLM to land within a few percent of it. Leave headroom on anything that fits by less than 2 GiB.
Throughput is not arithmetic. Tokens per second and time to first token depend on your engine version, batching and scheduling policy, prefix cache hit rate, chunked prefill settings, interconnect topology, and the shape of your actual traffic. Anyone who gives you a tokens-per-second figure from a dropdown is guessing, and we would rather say so than join them.
Two more things this deliberately leaves out: LoRA adapters and speculative decoding both add memory that depends on choices you have not made yet.
Method
How these numbers are derived.
All of it, including the parts that are estimates and the parts that are not.
The arithmetic
Weights first: the parameter count, minus the embedding and LM head that mainstream quantisers leave at 16-bit, converted at the precision you chose, plus what the quantisation scales themselves cost.
Then the KV cache: bytes per token per layer from num_kv_heads and head_dim, not num_attention_heads, multiplied by how many tokens of context actually get cached, since sliding-window layers stop growing past their window. Concurrency is what is left of GPU memory after weights, divided by that per-sequence cost.
That is the shape of it, not the full formal derivation. The exact equations, worked with real numbers for the model and GPU you actually pick, are not printed here as a static formula. They are the “show the arithmetic” panel under every result above. Expand one.
What is exact
The memory arithmetic. Parameter counts, layer counts, KV head counts, head dimensions and vocabulary sizes are read from each model's published config.json, not estimated. Device memory is the figure nvidia-smireports, which is below the number on the box. An “80GB” H100 reports 81559 MiB, and sizing against 80 × 1024 MiB is how people end up 400 MiB short at load time.
What is a modelled assumption
Two things, both specific to vLLM, because that is the engine this arithmetic models and the engine SovFleet runs today. First, the reserve for activations, CUDA context and CUDA graphs, which is modelled as vLLM's --gpu-memory-utilizationbecause that is exactly how vLLM behaves. It profiles the real activation peak at startup and caps itself at that fraction. Second, the overhead for quantisation scales and zero points, taken as group-128 with an fp16 scale: 0.156 bits per weight at 4-bit, 0.125 at 8-bit. An engine with a different memory model, if we add one, gets its own arithmetic here rather than being forced through vLLM's.
What is deliberately absent
Throughput. Tokens per second and time to first token are empirical, and depend on engine version, scheduling policy, prefix cache hit rate, chunked prefill configuration, interconnect topology and the shape of your traffic. We would rather leave the box empty than fill it with a number you cannot hold us to.
Also absent: LoRA adapter memory, speculative decoding draft models, multimodal vision towers, and CUDA graph capture overhead. Each depends on configuration you have not chosen yet.
Where it will be wrong
Real vLLM will usually report a KV cache within a few percent of these figures, sometimes slightly larger. Specific quantised checkpoints vary: some quantise the LM head, some leave extra layers at 16-bit, and a GPTQ checkpoint with a different group size will differ. If you have measured a real number that disagrees with this page, we would genuinely like to hear it.
Concurrency is not users
Whitepaper 03 · Technical
How Many Users Per GPU?
The question has no answer as asked, and a precise one once you fix two variables. Everything here is computable from a model's published configuration. No benchmarking, and no vendor required.
Head of Infrastructure, platform engineering, ML engineering · 12 minutes read
Reference
KV cache cost per token, by model.
At fp16 cache, across all layers. This is the number that decides concurrency, and it has almost nothing to do with parameter count. Qwen2.5 7B has four KV heads and costs less per token than models ten times its size.
| Model | Params | Layers | Attn heads | KV heads | head_dim | KV / token | KV / 1k tokens |
|---|---|---|---|---|---|---|---|
| Llama 3.1 8B Instructmeta-llama/Llama-3.1-8B-Instruct | 8.03B | 32 | 32 | 8 | 128 | 128 KiB | 128 MiB |
| Llama 3.2 3B Instructmeta-llama/Llama-3.2-3B-Instruct | 3.21B | 28 | 24 | 8 | 128 | 112 KiB | 112 MiB |
| Llama 3.3 70B Instructmeta-llama/Llama-3.3-70B-Instruct | 70.55B | 80 | 64 | 8 | 128 | 320 KiB | 320 MiB |
| Llama 3.1 405B Instructmeta-llama/Llama-3.1-405B-Instruct | 405.85B | 126 | 128 | 8 | 128 | 504 KiB | 504 MiB |
| Mistral 7B Instruct v0.3mistralai/Mistral-7B-Instruct-v0.3 | 7.25B | 32 | 32 | 8 | 128 | 128 KiB | 128 MiB |
| Mistral Small 3.1 24Bmistralai/Mistral-Small-3.1-24B-Instruct-2503 | 23.57B | 40 | 32 | 8 | 128 | 160 KiB | 160 MiB |
| Mixtral 8x7B Instructmistralai/Mixtral-8x7B-Instruct-v0.1 | 46.7B12.9B active | 32 | 32 | 8 | 128 | 128 KiB | 128 MiB |
| Qwen2.5 7B InstructQwen/Qwen2.5-7B-Instruct | 7.62B | 28 | 28 | 4 | 128 | 56 KiB | 56 MiB |
| Qwen2.5 32B InstructQwen/Qwen2.5-32B-Instruct | 32.76B | 64 | 40 | 8 | 128 | 256 KiB | 256 MiB |
| Qwen2.5-Coder 32B InstructQwen/Qwen2.5-Coder-32B-Instruct | 32.76B | 64 | 40 | 8 | 128 | 256 KiB | 256 MiB |
| Qwen2.5 72B InstructQwen/Qwen2.5-72B-Instruct | 72.7B | 80 | 64 | 8 | 128 | 320 KiB | 320 MiB |
| Qwen3 32BQwen/Qwen3-32B | 32.76B | 64 | 64 | 8 | 128 | 256 KiB | 256 MiB |
| Qwen3 30B-A3BQwen/Qwen3-30B-A3B | 30.5B3.3B active | 48 | 32 | 4 | 128 | 96 KiB | 96 MiB |
| Qwen3 235B-A22BQwen/Qwen3-235B-A22B | 235.1B22B active | 94 | 64 | 4 | 128 | 188 KiB | 188 MiB |
| DeepSeek-R1-Distill-Qwen 32Bdeepseek-ai/DeepSeek-R1-Distill-Qwen-32B | 32.76B | 64 | 40 | 8 | 128 | 256 KiB | 256 MiB |
| DeepSeek-V3 / R1 671Bdeepseek-ai/DeepSeek-V3 | 671B37B active | 61 | n/a | MLA | n/a | 69 KiB | 69 MiB |
| gpt-oss-20bopenai/gpt-oss-20b | 20.9B3.6B active | 24 | 64 | 8 | 64 | 48 KiB | 48 MiBwindowed layers cost less |
| gpt-oss-120bopenai/gpt-oss-120b | 116.8B5.1B active | 36 | 64 | 8 | 64 | 72 KiB | 72 MiBwindowed layers cost less |
| Gemma 3 27B ITgoogle/gemma-3-27b-it | 27.4B | 62 | 32 | 16 | 128 | 496 KiB | 496 MiBwindowed layers cost less |
| Phi-4 14Bmicrosoft/phi-4 | 14.7B | 40 | 40 | 10 | 128 | 200 KiB | 200 MiB |
| Sarvam-1sarvamai/sarvam-1 | 2B | 28 | 16 | 8 | 128 | 112 KiB | 112 MiB |
| Krutrim-2 Instructkrutrim-ai-labs/Krutrim-2-instruct | 12.2B | 40 | 32 | 8 | 128 | 160 KiB | 160 MiB |
Figures assume full attention on every layer. Models with sliding windows, Gemma 3 and gpt-oss here, cost the amount shown only up to the window length on their local layers, after which those layers stop growing. DeepSeek-V3 uses multi-head latent attention, which caches a compressed 512-wide latent plus a 64-wide RoPE key per layer rather than full K and V tensors.
Questions about GPU sizing
- How many concurrent users can an H100 serve?
- For Llama 3.1 8B in bf16 on a single 80GB H100: the weights take about 15 GiB, vLLM's default gpu-memory-utilization of 0.90 reserves about 8 GiB, and roughly 57 GiB is left for the KV cache. That model costs 128 KiB of cache per token, so an 8k-context sequence needs 1 GiB and about 56 fit at once. At 32k context the same card holds about 14. Concurrency is set by context length far more than by the model's parameter count.
- Why does the KV cache use num_kv_heads rather than num_attention_heads?
- Grouped-query attention shares one key-value head across several query heads. Llama 3.1 70B has 64 attention heads and 8 KV heads, and only the 8 are cached. Sizing the cache off the attention head count overstates it eightfold, which is the single most common error in back-of-envelope GPU sizing.
- How do I size a mixture-of-experts model?
- Memory follows total parameters and compute follows active parameters. Every expert must be resident in GPU memory even though only a couple run per token, so Qwen3-235B-A22B needs the memory footprint of a 235B model while decoding at roughly the speed of a 22B one. Budgeting from the active count is how people end up with a model that will not load.
- Does this tell me tokens per second?
- No, deliberately. Memory feasibility is arithmetic and can be stated exactly. Throughput is empirical: it depends on engine version, batching and scheduling policy, prefix cache hit rate, chunked prefill configuration, interconnect topology and your actual traffic shape. A dropdown that produces a tokens-per-second figure is guessing, and we would rather be useful about the part that can be known.
- How much memory does the KV cache use per token?
- 2 × num_kv_heads × head_dim × bytes_per_element, per layer, multiplied by the layer count. For Llama 3.1 8B that is 2 × 8 × 128 × 2 = 4096 bytes per layer, times 32 layers, which is 128 KiB per token. Multi-head latent attention models such as DeepSeek-V3 use a different formula and are roughly an order of magnitude smaller.
- Does fp8 KV cache double my concurrency?
- Close to it, since it halves the bytes per element. It is not free. It changes attention numerics and needs Ada, Hopper or Blackwell hardware. Measure quality on your own evaluations before shipping it rather than treating it as a pure win.
Now the harder question: who gets that capacity?
The arithmetic above tells you what your fleet can hold. It does not tell you what happens when three teams want it at once. That is the part we built.
An engineer replies within one working day. No qualification call before the technical one.