Hybrid inference platform
Managed inference, anywhere your GPUs run.
SovFleet gives you a control plane, a routing gateway and a node agent. Models serve wherever your GPUs already are, one data center, several clouds, or all of it treated as one fleet. Weights, KV cache and inference execution never leave the machine they run on.
The calculator is not gated. No form, no email, no call.
llama-3.1-70b
87 / 128 sequence slots
gpu-00
21 / 32
gpu-01
28 / 32
gpu-02
14 / 32
gpu-03
24 / 32
- serving
- saturated, queuing
- free
The allocation view, moving. Illustrative numbers, not a benchmark.
The problem
One model service is a project. Five is an operating model.
Serving is the solved part. What arrives later is capacity allocation between teams, quota, placement, and the evidence your auditors will ask for.
- 011 model · 1 team
A project
vLLM behind a load balancer. One command, one owner, one port. For one model and one owner, this is the correct architecture.
- 022 models · 2 teams
A negotiation
The second team turns every capacity question into an escalation, because there is no shared unit to have the conversation in. It gets settled by seniority rather than by demand.
- 035 models · 3 teams
A constraint problem
Tensor-parallel groups, memory arithmetic and compliance labels, re-solved by hand every time anything changes. Hand-solved constraints drift silently.
- 04the owner resigns
Key person risk
The engine flags live in one engineer's shell history. This is concentration risk in an operational system, and the kind your risk function finds during review rather than before it.
Each of these is solvable in-house. None of them ends. Each is a permanent fraction of an engineer, not a project with a completion date. The honest comparison, including what DIY does better.
Multi-tenancy
Teams self-serve. You stop being the ticket queue.
A team gets a key, a quota, and a set of models it is allowed to call. It deploys against that without asking you. When it hits the ceiling it sees its own ceiling rather than a stack trace, and the request that reaches you is a capacity conversation instead of a debugging session.
Quota is expressed in the unit that actually constrains you: concurrent sequence slots, not tokens per month. A GPU either has room for another sequence or it does not. Pretending otherwise is what makes token quotas feel arbitrary to the people they are applied to.
When the fleet fills, requests queue by a policy you set, priority, fair share, or hard reservation per team, and every team can see where it sits in that.
8 × H100 · llama-3.1-70b · 128 sequence slots
platform
29 / 32 slots
fraud-ml
48 / 48 slots
support-tooling
9 / 24 slots
unreserved
4 / 24 slots
- in use
- at reservation ceiling
- free
Illustrative layout of the allocation view. The numbers are an example, not a benchmark.
How it works
Three components. One of them runs on your machines.
Control plane
Runs in our cloud. Holds the fleet inventory, the model catalogue, team quotas and policy. It decides what should be running where. It never sees a prompt.
Gateway
Runs in the cloud and region you choose, dedicated to your account. Authenticates the key, resolves it to a team, applies that team's quota, and routes to a healthy node. Prompts and responses pass through in memory and are not persisted.
Node agent
Runs on your machines. Pulls weights from a registry you nominate, supervises the engine process, reports health, and opens the only network connection in the system, outbound.
Integration
Three lines change.
SovFleet speaks the OpenAI API. Your developers keep their client, their retries, their streaming code and their tests, they change a base URL and a key.
Tool calls, structured outputs and the embeddings endpoint behave as they already do.
from openai import OpenAI
client = OpenAI(
base_url="https://acme.sovfleet.dev/v1",
api_key="sf_...",
)
response = client.chat.completions.create(
model="llama-3.1-70b",
messages=[{"role": "user", "content": "..."}],
)Capacity calculator
What can the GPUs you already have actually serve?
Pick your hardware. It works out what fits, at what precision, across how many GPUs, and how many concurrent sequences the leftover memory holds, using the same arithmetic vLLM does at startup.
Open the calculatorNot gated. No form, no email, no call.
4 × H100 SXM 80GB
Illustrative. The calculator shows precision, placement and concurrent sequence counts for any fleet you enter.
Knowing when to buy
The number that gets the next purchase order signed.
“The GPUs are busy” does not get a purchase order signed. A queue depth over time does, alongside a rejection rate against a stated quota and a named team whose work is being delayed by a specific number of minutes a day.
SovFleet records admitted requests, queued requests, queue wait and preemptions, per team and per model. When you go to buy hardware, the argument is a chart of demand you turned away rather than an assertion that things feel slow.
It works in the other direction too: if the fleet is mostly idle, the same view says so, and the honest recommendation is that you should not buy anything yet.
Whitepaper 01 · Public sector banks
The Day After the GPUs Arrive
EASE 9.0 asks public sector banks to formulate a GPU strategy. Most of the difficulty is not in the procurement, it is in the operating model nobody has been asked to specify.
CIO, CTO, Head of Infrastructure, EASE programme office · 14 minutes read
Questions this audience asks first
- Does this run on GPUs we already own?
- That is the entire premise. SovFleet does not sell or resell GPU capacity. The node agent installs on whatever GPU machines you already have, one data center, several cloud accounts, a colo, or any mix of them, and serves models from hardware you have already paid for. SovFleet does not care where the machine sits, only that it is yours to run software on.
- How much do our developers have to change?
- The base URL and the API key. SovFleet presents an OpenAI-compatible API, so any client that already speaks to OpenAI, Azure OpenAI or another compatible endpoint points at it by changing configuration rather than code.
- Do we need to open inbound firewall ports?
- No. The node agent opens an outbound connection to the control plane, and the gateway reaches it over that connection. There are no inbound firewall rules to request and no VPN to run.
- How many concurrent users can one H100 serve?
- It depends on the model and the context length, and it is arithmetic rather than opinion. A single 80GB H100 running Llama 3.1 8B in bf16 has roughly 57 GiB left for the KV cache after weights and framework overhead, and that model costs 128 KiB of cache per token, so at 8k context it holds around 56 concurrent sequences. The capacity calculator on this site does that arithmetic for any fleet and shows its working.
Twenty minutes, no install
A screen-share walkthrough of the architecture with an engineer, before anyone touches your infrastructure. If you want to see it on your own hardware after that, the agent goes on a lab node next.
An engineer replies within one working day. No qualification call before the technical one.