SHARD Gateway
A reverse proxy that sits in front of vLLM and makes a real admission decision, admit, defer, or reject, before a request ever reaches the GPU.
The problem, and what SHARD Gateway actually does.
The problem
LLM inference servers run out of KV-cache memory under real concurrent load. Even before an outright crash, every request in flight starts queuing behind an unbounded FIFO, teams see multi-minute time-to-first-token under load, with zero visibility into who is actually waiting and why. Over-provisioning GPU capacity to stay safe is expensive; running close to the edge means every traffic spike degrades the whole fleet, not just the requests that caused it.
The solution
SHARD Gateway sits in front of vLLM and makes a real admission decision using live KV-cache telemetry. Under sustained overload it protects the requests it does admit instead of letting everyone wait in one unbounded queue. It fails open: if SHARD itself has a problem, traffic flows to vLLM exactly as if SHARD were never installed.
How a request flows through SHARD Gateway
If SHARD Gateway has a problem, it fails open, traffic flows straight through to vLLM untouched.
What this means for your GPU bill
If you resell GPU-hours, every SLA-violating or timed-out request on hardware you already paid for is lost or refunded revenue; SHARD converts more of your existing fleet into billable, SLA-compliant capacity, no new hardware required. If you pay for GPU capacity to run your own product, the usual alternative to admission control is over-provisioning. Sustaining the same SLO-compliance rate at higher utilization directly reduces how many extra nodes you need to buy. Illustrative math, not a promise: on an 8-GPU-node fleet at ₹150–400/GPU-hour running near-continuously, even a 20–30% reduction in the nodes needed to hit the same SLO target is roughly ₹25L–₹70L/year in avoided spend. Your real number depends on your real workload; the Diagnostic engagement measures it before any commitment.
The full evidence, generated 2026-07-21
Every number below is from a real, repeatable GCP run against a real vLLM instance — no simulation, no projection presented as measured fact.
Test environment
| Instance | GCP g2-standard-32 (32 vCPU, 128GB RAM) |
| GPU | 1× NVIDIA L4, 24GB HBM |
| Host OS | Debian 12 (bookworm) |
| Inference engine | vLLM v0.24.0, pinned |
| Serving container | vllm/vllm-openai:latest, Docker |
| Install-time gate | Full ctest suite run before any GPU spend |
Models under test
| Model | Params | Layers |
|---|---|---|
| OPT-1.3B | 1.3B | 24 |
| OPT-6.7B | 6.7B | 32 |
| Qwen2.5-7B-Instruct | 7B | 28 |
| Yi-9B | 9B | 48 |
What each phase actually tests
Moderate load — the realistic case
64-way concurrency, 200 prompts, a GPU-memory budget any sane operator would actually run in production. This is what a well-provisioned fleet looks like day to day.
Aggressive overload
128-way concurrency, 400 prompts, same memory budget as Phase A, pushed past what it can comfortably serve — real demand exceeding real capacity.
Connection-backlog burst
A fast, GPU-free TCP-stack test confirming the Gateway's own listen backlog holds under a connection burst.
Tight memory budget
Same load pattern as Phase B, but the GPU-memory-utilization ceiling is deliberately lowered — the lever that actually shrinks the real KV-cache pool.
OOM hunt — the extreme case
gpu-memory-utilization pushed to 0.97 with a 256-way concurrent burst. Built for one purpose: find a real CUDA out-of-memory crash.
Deterministic response-quality check
Real prompts at temperature=0, sent through both paths and diffed exactly — idle and under real concurrent contention. Does the Gateway ever alter what the model actually said?
No sane company runs production traffic at Phase D or E’s settings. A 0.16–0.97 GPU-memory-utilization ceiling with 128–256-way concurrency is not a realistic operating point — it’s a deliberately manufactured worst case, built specifically to try to force a real CUDA out-of-memory crash and see whether admission control prevents it.
Here is the honest result of that search: within everything we’ve been able to test so far, we have not produced a single OOM crash — not in baseline, not through the Gateway, on any of the four models, at any budget down to 0.16 utilization. We are not claiming a crash is impossible. We’re stating precisely what we found: in the scope we could check, it didn’t happen. The container-survival record below is the literal evidence for that claim, not a summary of it.
Phase A — at realistic load, the Gateway costs nothing
This is the headline, not Phase B/D/E below. At a budget any real deployment would run, the Gateway matches or beats raw vLLM’s own throughput — the admission layer is invisible until it’s actually needed.
Throughput, baseline vs. Gateway
OPT-6.7B and OPT-1.3B were not re-benchmarked at Phase A this cycle (prior clean baselines were reused to avoid redundant GPU spend) — no fresh number to report for them here.
Phase B — aggressive overload: same useful outcomes, delivered far faster
Completion count drops under the Gateway here, and that’s not the story: admission control rejects fast and honestly instead of letting every request queue for minutes toward a response nobody’s still waiting for. The number of requests that actually met the 60s SLO stays roughly the same on both sides — it just arrives 2-4x sooner.
Median latency, baseline vs. Gateway
Qwen’s Phase B predates the current admission fix and is excluded here pending re-verification, rather than shown as current.
Phase D — tight memory budget: the same pattern holds
Same story, different lever: instead of pushing more load at a fixed budget, the budget itself is pulled down toward the model’s real weight footprint.
Median latency, baseline vs. Gateway
Phase E — the OOM hunt: zero crashes found, anywhere we tested
0.97 utilization, 256-way burst — the closest this project has come to a deliberate crash test. Every container, every model, every phase: survived.
Median latency, baseline vs. Gateway
vLLM failed to boot at 0.97 util — real evidence of a floor, not a bug
Response-quality verification — does the Gateway ever change what the model said?
Real prompts, temperature=0, diffed exactly between direct vLLM and Gateway-fronted calls — idle and under real concurrent contention. 10 prompts each way, per model.
Every idle-phase check across all three models matched byte-for-byte — 30/30. That’s the direct proxy-transparency proof: the Gateway never touches tokens after admission. The two Qwen mismatches happened only under real injected background load, which is the signature of vLLM’s own preemption/recompute recovery, not the Gateway — a request preempted for KV-cache pressure and resumed can hit floating-point non-determinism across a different batch composition, occasionally flipping a near-tied token even at temperature=0. Both completions were coherent, on-topic, and correct — wording drift, not hallucination.
Two supported install paths.
Every step here is drawn directly from the project's own verified installer scripts, nothing here is aspirational or untested.
Path 1: Docker
Recommended if vLLM already runs in a container.
# Build the image
ZHERALDD_DIR=/path/to/zheraldd bash gateway/docker/build.sh
# Run it directly
docker run --rm -p 9100:9100 shard-gateway:latest http://<vllm-host>:8000 9100
# Or use the reference compose file
cd gateway/docker && docker compose upPath 2: Bare metal
Recommended if vLLM already runs bare-metal on the same host, or your security policy doesn’t allow containers on the inference host.
sudo bash gateway/docker/install.shThis builds from source, runs the real test suite as an install-time quality gate, installs to /opt/shard/, and sets up a systemd service. The service is disabled by default; edit the generated unit’s ExecStart with your real vLLM URL before starting it.
Prerequisites
- A Linux host (bare-metal path) or Docker (container path) with access to your vLLM instance
- An NVIDIA GPU already running vLLM, SHARD Gateway sits in front of it, it does not replace it
- For bare-metal installs: gcc, g++, cmake, ninja-build, git, and libsodium-dev (installed automatically via apt on Debian/Ubuntu)
- A full release tarball with this repo and its zheraldd build dependency as sibling directories, ask Solipher Labs if you don't already have it
The Gateway’s real CLI contract
shard_gateway <upstream_base_url> <listen_port> [poll_interval_ms] [shard_config_json] [num_layers]| upstream_base_url | Your real vLLM instance, e.g. http://127.0.0.1:8000 |
| listen_port | Port the Gateway itself listens on, e.g. 9100 |
| poll_interval_ms | How often the Gateway polls vLLM's own Prometheus metrics (default 500ms) |
| shard_config_json | HBM budget config, must roughly match vLLM's own --gpu-memory-utilization setting |
| num_layers | Your model's real transformer layer count, feeds the Gateway's per-request KV-byte estimate |
Environment variables
| SHARD_LICENSE_PATH | Path to a signed license file. Unset = license verification skipped entirely. |
| SHARD_GATEWAY_AUTH_TOKEN | Shared-secret bearer token required on every route except GET /health. Empty by default. |
Verifying the install
Both install paths run the real gateway/shard ctest suite as a hard gate before completing. Check GET /healthon the Gateway’s listen port once running, a healthy Gateway returns 200; a degraded one returns 503 and automatically fails open. Send one real request through the Gateway’s port and confirm you get a normal vLLM response before pointing real traffic at it.
Known limitations, honestly
Neither install path is a one-command “download and go” experience yet, both need the zheraldd dependency provided as a separate source tree. If you’re installing this yourself rather than through Solipher directly, ask for a full release tarball that bundles both. A mismatched shard_config_json / num_layers relative to your real vLLM startup flags makes admission decisions meaningless, double-check these before production.
Start with the free tier, self-hosted.
First-draft India-market pricing, ranges rather than fixed numbers, intended as a starting point for discussion.
Community
Free
libshard core, self-hosted, MIT-licensed core, community support.
Diagnostic
₹50,000 – ₹1,50,000 (one-time)
1-week hands-on engagement: a Solipher engineer measures your real baseline, root-causes the problem, and delivers a projected-savings report.
Growth
Popular₹6,000 – ₹12,000 / GPU-node / month (billed annually)
Production use, up to 8 GPU nodes, signed-license enforcement, business-hours support, 48hr bug-fix SLA.
Enterprise
₹15L – ₹40L+ / year, custom
Unlimited nodes, 24/7 support, dedicated support channel, custom SLA, hash-chained admission audit certificates for regulated industries.
Get the release tarball
SHARD Gateway is in active pilot testing and isn’t a self-contained public download yet. Email us and we’ll send the full release tarball (Gateway plus its build dependency) and get your 14-day trial started, no card required.
SHARD Gateway is in active pilot testing. Enforce-mode certification against vLLM’s own scheduler integration surface is not yet complete (Capability Audit v0.24.0), SHARD’s proven path today is the reverse-proxy admission-control model described above, which does not require it.
Looking for the context-compilation side of this work? See Solipher SHARD Context on the Products page.