SovFleet
Request a demo

Product

What installs where, and what each part can see.

Three components. Two of them run in our cloud and hold no model data. One runs on your machine and holds all of it. The rest of this page is the detail behind that sentence.

SovFleet architecture and trust boundaryArchitecture of SovFleet. Left to right, the inference path runs: your applications, calling an OpenAI-compatible endpoint; the SovFleet gateway, a dedicated instance running in the cloud and region you chose for your account, which authenticates the key, applies the team's quota and picks a node; the node agent, running on your own infrastructure, which receives the request over a connection the agent itself opened outbound; and your GPU nodes, where the inference engine, vLLM today, holds the model weights and the KV cache and executes the forward pass. A separate dashed line runs from the node agent back to the SovFleet control plane carrying telemetry only: GPU health, utilisation metrics, token counts, and endpoint and model names. A heavy border encloses the node agent and the GPU nodes, marking your trust boundary: model weights, KV cache and inference execution never cross it. Prompts and responses do cross it, in transit through your dedicated gateway instance, where they are held in memory for the life of the request and never written to disk.SOVFLEET CLOUD · DEDICATED, YOUR REGIONYOUR TRUST BOUNDARYweights · KV cache · execution never leaveYour appsOpenAI-compatibleclient, 3 linesof config changedGatewayAuthenticates the keyApplies team quotaPicks a healthy nodeNode agentYour host, your VLANOutbound connectiononly. No inbound ruleGPU nodeInference engineWeights + KV cacheForward passpromptTLSpromptagent-initiated tunnelpromptlocalPrompts and responses transit hereheld in memory, never written to diskControl planeFleet inventoryQuota and policyMetrics and alertingtelemetry, outbound onlyGPU health · utilisation · token countsendpoint and model names21 of 32 sequence slots in useSolid blue: the inference path. Dashed green: telemetry. Nothing crosses the boundary right to left except responses.A customer-hosted gateway, which removes the transit hop entirely, is on the roadmap and is not shipped today.
The gateway is drawn outside your boundary because that is where it runs today. See security and data residency for what that means in review.

Whitepaper 04 · Operating model

Sovereign by Architecture

Owning the hardware is the easy half. This is what the operating model above it has to do, capacity allocation, governance, evidence, and how SovFleet implements it.

CIO, Head of Infrastructure, Head of Data Science, CISO · 15 minutes read

One email with the link. No sequence, no newsletter.

Component by component

Each SovFleet component, where it runs, what it can see and what network access it needs.
ComponentRunsCan seeNever seesNetwork
Control planeSovFleet cloudFleet inventory, model catalogue, quotas, metrics, audit logPrompts, responses, weights, KV cacheAccepts the agent's outbound connection
GatewaySovFleet cloudAPI keys, request and response bodies in memory, token countsWeights, KV cache; nothing written to diskAccepts client HTTPS; reuses the agent's connection outbound
Node agentYour machineEverything on that host: weights, prompts, responses, KV cacheOther customers' anything. It is a single-tenant processOutbound 443 only. No listener, no inbound rule
Engine (vLLM today)Your machineWeights, KV cache, the forward passAny network destination outside your hostLocalhost only

The request path

Four hops, two of them inside your network.

ClientPOST /v1/chat/completionsGatewaykey → team → quota → nodeAgentover its own outbound tunnelvLLMprefill, then decodeClienttokens stream backSteps 1 and 2 cross the public internet. Steps 3 and 4 are inside your network.

  1. 1

    Client to gateway

    Standard HTTPS to your subdomain. The request is an OpenAI-format chat completion; nothing about it is proprietary to us.

  2. 2

    Gateway resolves and routes

    The key resolves to a team. The team's quota is checked, the model name is resolved to a running deployment, and a node with a free sequence slot is chosen. If none is free, the request queues or is rejected according to your policy.

  3. 3

    Gateway to node agent

    Over the connection the agent opened outbound. There is no inbound listener on your side and nothing to allow through a firewall.

  4. 4

    Agent to engine

    Localhost to the engine process, vLLM today. Prefill, then decode. Tokens stream back along the same path, and the KV cache stays where it was built.

On latency

Two of those hops cross the public internet, and that costs you round-trip time. How much depends entirely on where your GPUs sit relative to the gateway region, so we do not publish a number. Anyone who does is quoting their own lab. Measure it on your own network during the trial; it is the first thing we help you set up.

Installation

One binary, one token, one config file.

The agent is a single static binary. It needs a host with the NVIDIA driver and container runtime you already use, outbound 443, and a registry it can pull weights from. That registry can be yours. An internal mirror, an S3-compatible bucket, or a filesystem path where the weights already sit.

It does not need root beyond what your container runtime already requires, and it does not install a kernel module.

Install is three steps on each GPU host: pull the release, enrol it against your account with a token issued from the console, and start it as a system service. Every release is GPG-signed with published checksums, so you verify what you are running before it touches a host that holds your weights. We walk through the actual commands on the call rather than publish a copy-pasted script here.

gpu-node-03 · enrolled

GPUs offered
0, 1, 2, 3
Zone label
dc-mum-1
Compliance label
pci
Weights registry
your S3-compatible mirror, not the public internet
Egress allowed
the control plane and your registry endpoint. Nothing else.

What one host looks like after enrolment. Every field here is something you set, not something we infer.

Placement

How a model ends up on a particular card.

You declare a deployment: a model, a precision, a context length, a target number of replicas, and which node labels it may land on. The control plane solves for placement using the same arithmetic the capacity calculator uses, weights first, then whether the remaining memory holds enough KV cache to be worth running.

Labels are how compliance boundaries get enforced in practice. A deployment marked compliance: pci is only ever placed on hosts carrying that label, and the scheduler will leave a GPU idle rather than violate it.

Tensor-parallel groups are placed on cards within a single host unless you explicitly allow otherwise, because cross-host TP over anything short of InfiniBand is usually a mistake.

deployment

int4-awq

llama-3.1-70b

32k context · TP ×4 · 2 replicas

Placement requires
zone: dc-mum-1 · compliance: pci
fraud-ml
48 reserved slots
support-tools
24 reserved slots
Overflow policy
fair share

What one deployment declares. The scheduler reads this the same way it reads every other one.

Failure

What breaks, and what happens when it does.

Every security review asks this, so here it is unprompted.

A GPU node dies

In-flight requests on that node fail and are not retried automatically, because a partially streamed completion cannot be safely replayed. New requests route to the remaining replicas. If that drops you below your reserved slots, the affected teams are told which ones.

The control plane is unreachable

Running deployments keep serving. The agent holds its last known configuration and continues. What you lose is the ability to change anything: new deployments, quota edits and placement decisions all wait.

The gateway is unreachable

Requests fail. This is the honest answer and the sharpest edge in the current architecture: there is no direct client-to-agent path today. It is the same failure surface as any hosted API, and it is the main thing the customer-hosted gateway on the roadmap is meant to remove.

Your registry is unreachable

Already-loaded models keep serving from GPU memory. New model pulls fail with the registry error rather than a generic one, because a scheduling failure that does not name the cause is worse than useless.

A team exhausts its quota

Requests beyond the reservation queue or receive a 429 with a Retry-After, depending on the policy you set. Other teams are not affected. That is the point of the reservation.

The fleet is genuinely full

Requests queue by policy. Queue depth and wait time are recorded per team, which is the evidence you will need if the answer turns out to be more hardware.

Scope

What this does not do.

  • It does not train or fine-tune. Serving only. Bring the weights you have.
  • It does not supply GPUs. There is no marketplace and no burst-to-our-cloud option. If your fleet is full, it is full, and we will show you the evidence for that rather than sell you capacity.
  • It does not write a custom inference engine.It supervises one, vLLM today, and the direction is toward the node agent treating the engine as a plugin rather than a fixed choice, so an engine like SGLang or TensorRT-LLM slots in the same way. That interface is not built yet. When today's engine ships a faster kernel you get it by upgrading, not by waiting for us to reimplement it.
  • It is not a RAG framework, an agent framework, or a prompt manager. It terminates an OpenAI-compatible API and puts tokens on GPUs.

Walk through this with an engineer first

Twenty minutes, screen share, no install. If it holds up, the next step is the agent on one lab node, with your own tooling watching where the traffic goes.

Request a demoSecurity and data residency

An engineer replies within one working day. No qualification call before the technical one.