Self-hosted LLM inference platform
Managed inference on the GPUs you already own.
Atlas gives you a control plane, a routing gateway and a node agent. Models serve from your hardware — on-premise, colocated, or in your own cloud account. Weights, KV cache and inference execution never leave it.
The calculator is not gated. No form, no email, no call.
from openai import OpenAI
client = OpenAI(
base_url="https://acme.atlas.example.com/v1",
api_key="atl_...",
)
response = client.chat.completions.create(
model="llama-3.1-70b",
messages=[{"role": "user", "content": "..."}],
)Streaming, tool calls, structured outputs and the embeddings endpoint behave as they already do in your code.
The problem
The first model on your own GPUs is a weekend. The fifth is a team.
Standing up vLLM behind a load balancer works. One model, one team, one person who understands the flags. Most organisations reading this have already done it, and they were right to.
What breaks is everything after that. A second team wants a different model. A third wants the same model with a longer context. Someone needs to know why their requests started queuing at 4pm and whether that is their fault. Finance wants to know which cost centre used the fleet last month. A researcher pins an engine version for a benchmark and now nobody dares upgrade it.
Then the person who understands the flags takes a job somewhere else.
None of this is a serving problem. vLLM serves, and it serves well. It is a multi-tenancy, quota, placement and observability problem, and it lands on the platform team as a permanent queue of tickets.
How it works
Three components. One of them runs on your machines.
Control plane
Runs in our cloud. Holds the fleet inventory, the model catalogue, team quotas and policy. It decides what should be running where. It never sees a prompt.
Gateway
Runs in our cloud. Authenticates the key, resolves it to a team, applies that team's quota, and routes to a healthy node. Prompts and responses pass through in memory and are not persisted.
Node agent
Runs on your machines. Pulls weights from a registry you nominate, supervises the engine process, reports health, and opens the only network connection in the system — outbound.
Capacity calculator
What can the GPUs you already have actually serve?
Pick your hardware. This works out what fits, at what precision, across how many GPUs, and how many concurrent sequences the leftover memory holds. It shows its working, and it is exact about the part that is arithmetic.
The window you intend to serve, not the model's maximum. Sizing for 128k when your requests are 4k wastes most of the cache.
Auto picks the highest precision that fits, working down from bf16.
fp8 KV halves cache memory and roughly doubles concurrency. It is not free — measure quality on your own evals before shipping it.
vLLM's --gpu-memory-utilization. The remainder holds the CUDA context, activations and CUDA graphs.
4 GPUs · 319 GiB of device memory · 4 × H100 SXM 80GB
You could run
4 × H100 SXM 80GB
319 GiB total · sm90 (Hopper) · 3,350 GB/s
| Model | Weights | Placement | Concurrent at 8k | Show arithmetic |
|---|---|---|---|---|
| llama-3.1-8b8.03B | bf1615.0 GiB | 1 GPU4 replicas | ~56224 across the group | |
| llama-3.3-70b70.55B | bf16131.4 GiB | TP ×41 replica | ~62concurrent | |
| mistral-7b-v0.37.25B | bf1613.5 GiB | 1 GPU4 replicas | ~58232 across the group | |
| qwen2.5-32b32.76B | bf1661.0 GiB | TP ×22 replicas | ~4182 across the group | |
| qwen3-30b-a3b30.5B total · 3.3B active | bf1656.8 GiB | 1 GPU4 replicas | ~1976 across the group | |
| gpt-oss-120b116.8B total · 5.1B active | bf16217.6 GiB | TP ×41 replica | ~242concurrent |
Concurrency is the number of sequences whose KV cache fits simultaneously at the full context length. Requests shorter than 8k use proportionally less, so real fleets run more than this. One sequence costs 1.0 GiB for the first model listed.
What this cannot tell you
Memory feasibility is arithmetic, and the arithmetic above is exact. If the numbers say a model fits, it fits.
Throughput is not arithmetic. Tokens per second and time to first token depend on your engine version, batching and scheduling policy, prefix cache hit rate, chunked prefill settings, interconnect topology, and the shape of your actual traffic. Anyone who gives you a tokens-per-second figure from a dropdown is guessing, and we would rather say so than join them.
Two more things this deliberately leaves out: LoRA adapters and speculative decoding both add memory that depends on choices you have not made yet.
Open the full calculator to pick from every model, change precision and share the result as a link.
Multi-tenancy
Teams self-serve. You stop being the ticket queue.
A team gets a key, a quota, and a set of models it is allowed to call. It deploys against that without asking you. When it hits the ceiling it sees its own ceiling rather than a stack trace, and the request that reaches you is a capacity conversation instead of a debugging session.
Quota is expressed in the unit that actually constrains you: concurrent sequence slots, not tokens per month. A GPU either has room for another sequence or it does not. Pretending otherwise is what makes token quotas feel arbitrary to the people they are applied to.
When the fleet fills, requests queue by a policy you set — priority, fair share, or hard reservation per team — and every team can see where it sits in that.
8 × H100 · llama-3.1-70b · 128 sequence slots
platform
29 / 32 slots
fraud-ml
48 / 48 slots
support-tooling
9 / 24 slots
unreserved
4 / 24 slots
- in use
- at reservation ceiling
- free
Illustrative layout of the allocation view. The numbers are an example, not a benchmark.
Knowing when to buy
The number that gets the next purchase order signed.
“The GPUs are busy” does not get a purchase order signed. A queue depth over time does, alongside a rejection rate against a stated quota and a named team whose work is being delayed by a specific number of minutes a day.
Atlas records admitted requests, queued requests, queue wait and preemptions, per team and per model. When you go to buy hardware, the argument is a chart of demand you turned away rather than an assertion that things feel slow.
It works in the other direction too, which is the part vendors usually leave out: if the fleet is mostly idle, the same view says so, and the honest recommendation is that you should not buy anything yet.
Where we actually are
Things we would rather you heard from us.
Prompts transit our gateway today
We hold no certifications yet
There is no customer logo row on this site
Questions this audience asks first
- Do our prompts leave our network?
- Today, yes — prompts and responses transit the Atlas gateway on their way to your node agent. They are held in memory for the life of the request and are never written to disk or used for training. Model weights, the KV cache and inference execution never leave your hardware. A customer-hosted gateway, where nothing transits Atlas at all, is on the roadmap and is not shipped today.
- Does this run on GPUs we already own?
- That is the entire premise. Atlas does not sell or resell GPU capacity. The node agent installs on your existing machines — on-premise, colocated, or in your own cloud account — and serves models from hardware you have already paid for.
- How much do our developers have to change?
- The base URL and the API key. Atlas presents an OpenAI-compatible API, so any client that already speaks to OpenAI, Azure OpenAI or another compatible endpoint points at it by changing configuration rather than code.
- Do we need to open inbound firewall ports?
- No. The node agent opens an outbound connection to the control plane, and the gateway reaches it over that connection. There are no inbound firewall rules to request and no VPN to run.
- How many concurrent users can one H100 serve?
- It depends on the model and the context length, and it is arithmetic rather than opinion. A single 80GB H100 running Llama 3.1 8B in bf16 has roughly 57 GiB left for the KV cache after weights and framework overhead, and that model costs 128 KiB of cache per token, so at 8k context it holds around 56 concurrent sequences. The capacity calculator on this site does that arithmetic for any fleet and shows its working.
See it on your own hardware
Thirty minutes. We install the agent on one of your nodes, point a client at it, and you watch where the traffic goes.
Priya Raghavan replies within one working day. No sequence, no SDR, no calendar tennis.