How many concurrent users can an H100 serve?

For Llama 3.1 8B in bf16 on a single 80GB H100: the weights take about 15 GiB, vLLM's default gpu-memory-utilization of 0.90 reserves about 8 GiB, and roughly 57 GiB is left for the KV cache. That model costs 128 KiB of cache per token, so an 8k-context sequence needs 1 GiB and about 56 fit at once. At 32k context the same card holds about 14. Concurrency is set by context length far more than by the model's parameter count.

Why does the KV cache use num_kv_heads rather than num_attention_heads?

Grouped-query attention shares one key-value head across several query heads. Llama 3.1 70B has 64 attention heads and 8 KV heads, and only the 8 are cached. Sizing the cache off the attention head count overstates it eightfold, which is the single most common error in back-of-envelope GPU sizing.

How do I size a mixture-of-experts model?

Memory follows total parameters and compute follows active parameters. Every expert must be resident in GPU memory even though only a couple run per token, so Qwen3-235B-A22B needs the memory footprint of a 235B model while decoding at roughly the speed of a 22B one. Budgeting from the active count is how people end up with a model that will not load.

How much memory does the KV cache use per token?

2 × num_kv_heads × head_dim × bytes_per_element, per layer, multiplied by the layer count. For Llama 3.1 8B that is 2 × 8 × 128 × 2 = 4096 bytes per layer, times 32 layers, which is 128 KiB per token. Multi-head latent attention models such as DeepSeek-V3 use a different formula and are roughly an order of magnitude smaller.

Does fp8 KV cache double my concurrency?

Close to it, since it halves the bytes per element. It is not free — it changes attention numerics and needs Ada, Hopper or Blackwell hardware. Measure quality on your own evaluations before shipping it rather than treating it as a pure win.

Capacity calculator

What can your GPUs actually serve?

Select the hardware you already have. This works out which models fit, at what precision, across how many GPUs, and how many concurrent sequences the memory left over will hold. Every number has an arithmetic derivation you can check.

No form, no email, no account. The results are useful whether or not you ever talk to us.

4 GPUs · 319 GiB of device memory · 4 × H100 SXM 80GB

You could run

4 × H100 SXM 80GB

319 GiB total · sm90 (Hopper) · 3,350 GB/s

Models that fit on 4 × H100 SXM 80GB at 8k context, with the concurrency each allows.
Model	Weights	Placement	Concurrent at 8k
llama-3.1-8b8.03B	bf1615.0 GiB	1 GPU4 replicas	~56224 across the group
llama-3.3-70b70.55B	bf16131.4 GiB	TP ×41 replica	~62concurrent
mistral-7b-v0.37.25B	bf1613.5 GiB	1 GPU4 replicas	~58232 across the group
qwen2.5-32b32.76B	bf1661.0 GiB	TP ×22 replicas	~4182 across the group
qwen3-30b-a3b30.5B total · 3.3B active	bf1656.8 GiB	1 GPU4 replicas	~1976 across the group
gpt-oss-120b116.8B total · 5.1B active	bf16217.6 GiB	TP ×41 replica	~242concurrent

Concurrency is the number of sequences whose KV cache fits simultaneously at the full context length. Requests shorter than 8k use proportionally less, so real fleets run more than this. One sequence costs 1.0 GiB for the first model listed.

What this cannot tell you

Memory feasibility is arithmetic, and the arithmetic above is exact. If the numbers say a model fits, it fits.

Throughput is not arithmetic. Tokens per second and time to first token depend on your engine version, batching and scheduling policy, prefix cache hit rate, chunked prefill settings, interconnect topology, and the shape of your actual traffic. Anyone who gives you a tokens-per-second figure from a dropdown is guessing, and we would rather say so than join them.

Two more things this deliberately leaves out: LoRA adapters and speculative decoding both add memory that depends on choices you have not made yet.

Method

How these numbers are derived.

All of it, including the parts that are estimates and the parts that are not.

The arithmetic

# Weights. Mainstream quantisers leave the embedding and
# LM head at 16-bit, which matters more than people expect
# on small models with large vocabularies.
embed_params  = vocab_size * hidden_size * (1 if tied else 2)
quant_params  = total_params - embed_params

weight_bytes  = embed_params * 2
              + quant_params * bits_per_weight / 8
              + quant_params * scale_bits / 8

# KV cache. num_kv_heads, NOT num_attention_heads.
kv_per_token_per_layer = 2 * num_kv_heads * head_dim * kv_bytes

# Sliding-window layers stop growing past the window.
tokens_cached = (global_layers * context
               + local_layers * min(context, window))

kv_per_sequence = kv_per_token_per_layer * tokens_cached

# What is left.
usable    = vram_bytes * gpu_memory_utilization
kv_pool   = usable * tensor_parallel - weight_bytes

concurrent = floor(kv_pool / kv_per_sequence)

What is exact

The memory arithmetic. Parameter counts, layer counts, KV head counts, head dimensions and vocabulary sizes are read from each model's published config.json, not estimated. Device memory is the figure nvidia-smireports, which is below the number on the box — an “80GB” H100 reports 81559 MiB, and sizing against 80 × 1024 MiB is how people end up 400 MiB short at load time.

What is a modelled assumption

Two things. First, the reserve for activations, CUDA context and CUDA graphs, which is modelled as vLLM's --gpu-memory-utilization because that is exactly how vLLM behaves — it profiles the real activation peak at startup and caps itself at that fraction. Second, the overhead for quantisation scales and zero points, taken as group-128 with an fp16 scale: 0.156 bits per weight at 4-bit, 0.125 at 8-bit.

What is deliberately absent

Throughput. Tokens per second and time to first token are empirical, and depend on engine version, scheduling policy, prefix cache hit rate, chunked prefill configuration, interconnect topology and the shape of your traffic. We could produce a plausible-looking number. It would be a guess wearing a monospace font.

Also absent: LoRA adapter memory, speculative decoding draft models, multimodal vision towers, and CUDA graph capture overhead. Each depends on configuration you have not chosen yet.

Where it will be wrong

Real vLLM will usually report a KV cache within a few percent of these figures, sometimes slightly larger. Specific quantised checkpoints vary: some quantise the LM head, some leave extra layers at 16-bit, and a GPTQ checkpoint with a different group size will differ. If you have measured a real number that disagrees with this page, we would genuinely like to hear it.

Concurrency is not users

This counts sequences whose KV cache fits simultaneously at the full context length. Real requests are usually shorter and free their cache when they finish, so a fleet serving interactive traffic supports many more people than this number. Treat it as the floor at worst case, not the ceiling.

Reference

KV cache cost per token, by model.

At fp16 cache, across all layers. This is the number that decides concurrency, and it has almost nothing to do with parameter count — Qwen2.5 7B has four KV heads and costs less per token than models ten times its size.

KV cache cost per token for each model in the catalogue, at fp16 cache precision.
Model	Params	Layers	Attn heads	KV heads	head_dim	KV / token	KV / 1k tokens
Llama 3.1 8B Instructmeta-llama/Llama-3.1-8B-Instruct	8.03B	32	32	8	128	128 KiB	128 MiB
Llama 3.2 3B Instructmeta-llama/Llama-3.2-3B-Instruct	3.21B	28	24	8	128	112 KiB	112 MiB
Llama 3.3 70B Instructmeta-llama/Llama-3.3-70B-Instruct	70.55B	80	64	8	128	320 KiB	320 MiB
Llama 3.1 405B Instructmeta-llama/Llama-3.1-405B-Instruct	405.85B	126	128	8	128	504 KiB	504 MiB
Mistral 7B Instruct v0.3mistralai/Mistral-7B-Instruct-v0.3	7.25B	32	32	8	128	128 KiB	128 MiB
Mistral Small 3.1 24Bmistralai/Mistral-Small-3.1-24B-Instruct-2503	23.57B	40	32	8	128	160 KiB	160 MiB
Mixtral 8x7B Instructmistralai/Mixtral-8x7B-Instruct-v0.1	46.7B12.9B active	32	32	8	128	128 KiB	128 MiB
Qwen2.5 7B InstructQwen/Qwen2.5-7B-Instruct	7.62B	28	28	4	128	56 KiB	56 MiB
Qwen2.5 32B InstructQwen/Qwen2.5-32B-Instruct	32.76B	64	40	8	128	256 KiB	256 MiB
Qwen2.5-Coder 32B InstructQwen/Qwen2.5-Coder-32B-Instruct	32.76B	64	40	8	128	256 KiB	256 MiB
Qwen2.5 72B InstructQwen/Qwen2.5-72B-Instruct	72.7B	80	64	8	128	320 KiB	320 MiB
Qwen3 32BQwen/Qwen3-32B	32.76B	64	64	8	128	256 KiB	256 MiB
Qwen3 30B-A3BQwen/Qwen3-30B-A3B	30.5B3.3B active	48	32	4	128	96 KiB	96 MiB
Qwen3 235B-A22BQwen/Qwen3-235B-A22B	235.1B22B active	94	64	4	128	188 KiB	188 MiB
DeepSeek-R1-Distill-Qwen 32Bdeepseek-ai/DeepSeek-R1-Distill-Qwen-32B	32.76B	64	40	8	128	256 KiB	256 MiB
DeepSeek-V3 / R1 671Bdeepseek-ai/DeepSeek-V3	671B37B active	61	—	MLA	—	69 KiB	69 MiB
gpt-oss-20bopenai/gpt-oss-20b	20.9B3.6B active	24	64	8	64	48 KiB	48 MiBwindowed layers cost less
gpt-oss-120bopenai/gpt-oss-120b	116.8B5.1B active	36	64	8	64	72 KiB	72 MiBwindowed layers cost less
Gemma 3 27B ITgoogle/gemma-3-27b-it	27.4B	62	32	16	128	496 KiB	496 MiBwindowed layers cost less
Phi-4 14Bmicrosoft/phi-4	14.7B	40	40	10	128	200 KiB	200 MiB

Figures assume full attention on every layer. Models with sliding windows — Gemma 3 and gpt-oss here — cost the amount shown only up to the window length on their local layers, after which those layers stop growing. DeepSeek-V3 uses multi-head latent attention, which caches a compressed 512-wide latent plus a 64-wide RoPE key per layer rather than full K and V tensors.

Questions about GPU sizing

How many concurrent users can an H100 serve?: For Llama 3.1 8B in bf16 on a single 80GB H100: the weights take about 15 GiB, vLLM's default gpu-memory-utilization of 0.90 reserves about 8 GiB, and roughly 57 GiB is left for the KV cache. That model costs 128 KiB of cache per token, so an 8k-context sequence needs 1 GiB and about 56 fit at once. At 32k context the same card holds about 14. Concurrency is set by context length far more than by the model's parameter count.
Why does the KV cache use num_kv_heads rather than num_attention_heads?: Grouped-query attention shares one key-value head across several query heads. Llama 3.1 70B has 64 attention heads and 8 KV heads, and only the 8 are cached. Sizing the cache off the attention head count overstates it eightfold, which is the single most common error in back-of-envelope GPU sizing.
How do I size a mixture-of-experts model?: Memory follows total parameters and compute follows active parameters. Every expert must be resident in GPU memory even though only a couple run per token, so Qwen3-235B-A22B needs the memory footprint of a 235B model while decoding at roughly the speed of a 22B one. Budgeting from the active count is how people end up with a model that will not load.
Does this tell me tokens per second?: No, deliberately. Memory feasibility is arithmetic and can be stated exactly. Throughput is empirical: it depends on engine version, batching and scheduling policy, prefix cache hit rate, chunked prefill configuration, interconnect topology and your actual traffic shape. A dropdown that produces a tokens-per-second figure is guessing, and we would rather be useful about the part that can be known.
How much memory does the KV cache use per token?: 2 × num_kv_heads × head_dim × bytes_per_element, per layer, multiplied by the layer count. For Llama 3.1 8B that is 2 × 8 × 128 × 2 = 4096 bytes per layer, times 32 layers, which is 128 KiB per token. Multi-head latent attention models such as DeepSeek-V3 use a different formula and are roughly an order of magnitude smaller.
Does fp8 KV cache double my concurrency?: Close to it, since it halves the bytes per element. It is not free — it changes attention numerics and needs Ada, Hopper or Blackwell hardware. Measure quality on your own evaluations before shipping it rather than treating it as a pure win.

Now the harder question: who gets that capacity?

The arithmetic above tells you what your fleet can hold. It does not tell you what happens when three teams want it at once. That is the part we built.

Request a demo How Atlas works

Priya Raghavan replies within one working day. No sequence, no SDR, no calendar tennis.