SGLang Explained: The Inference Engine Eating the Stack

SGLang runs inference for Google, Microsoft, NVIDIA, Oracle, AMD, LinkedIn, and xAI — generating trillions of tokens per day across more than 400,000 GPUs as of 2026, per nextomoro’s SGLang profile. That’s not a typo. A single open-source serving framework, initially released in 2024 and developed by Ying Sheng and Banghua Zhu out of LMSYS (the Berkeley research collective behind Chatbot Arena), has become the default inference substrate for the largest AI deployments on the planet.

Here’s why that matters for you: the inference engine you pick determines whether your GPU fleet runs at 60% or 95% utilization, whether your multi-turn agent pipeline recomputes KV cache on every request or reuses 80% of it, and whether your monthly bill stays predictable or quietly doubles from hidden costs. SGLang’s commercial backer, RadixArk, launched in May 2026 with a $100 million seed round at a $400 million valuation, but the framework itself remains Apache 2.0 and free for self-hosting.

What I find more significant than the valuation is a structural inversion I’ve been watching: hardware vendors now optimize for SGLang compatibility first. AMD, Moore Threads, and Intel are all building backends that merge into SGLang’s mainline. The framework has become the ecosystem chokepoint — the layer where hardware meets software — and that position compounds. If you’re evaluating inference engines today, you need to understand both what SGLang does well and where it’ll cost you.

What SGLang Actually Does: RadixAttention and Structured Generation

SGLang’s core technical bet is RadixAttention — a radix-tree-based mechanism for caching and reusing prefix portions of language model computation across requests, as documented in the original SGLang research. Where vLLM’s PagedAttention manages KV cache at the block level using hash-based lookup, SGLang stores cached attention activations in a token-level radix tree that automatically discovers and reuses shared prefixes. The practical impact: token-level prefix caching can reduce input token costs by 40-60% in multi-turn conversations, compared to vLLM’s block-level approach.

The second pillar is SGLang’s structured-program-execution model. Rather than treating structured generation as an external add-on, SGLang compiles structured generation patterns — regex-constrained outputs, structured JSON, agentic control flow — into efficient runtime sequences at the scheduler level. vLLM relies on external tools like Outlines for the same functionality, which introduces overhead at high batch sizes. If you’re building agentic pipelines with tool calls, conditional branching, or multi-step reasoning, this matters: the runtime understands your workflow shape, not just individual token sequences.

SGLang is not a foundation model. It runs models from other labs — DeepSeek, Llama, Qwen, Kimi, GLM, GPT-OSS, Gemma, and Mistral families all work out of the box. It exposes OpenAI-compatible endpoints (/v1/chat/completions and /v1/completions, default port 30000), so migrating from OpenAI’s API is a configuration change, not a rewrite. The server applies Hugging Face tokenizer chat templates when available, and supports streaming, embeddings, reasoning parsers for DeepSeek/Qwen3/Kimi/GPT-OSS, and optional API-key auth.

SGLang vs vLLM: Where Each Engine Actually Wins

The throughput gap between SGLang and vLLM depends heavily on your workload shape, and the numbers tell a more nuanced story than vendor blogs suggest. On H100 80GB with Llama 3.1 8B, SGLang achieves approximately 16,200 tokens per second — roughly 29% faster than vLLM’s 12,500 tok/s. On A100 80GB with the same model, SGLang hits 6,800 tok/s versus vLLM’s 6,200 tok/s. The gap widens on smaller models because prefill is a larger fraction of total compute, and RadixAttention’s prefix caching pays off more.

Scale up to Llama 3.3 70B Instruct at FP8 with 50-way concurrency, and the delta shrinks to 1,920 tok/s for SGLang versus 1,850 for vLLM — just 3.8%. At 70B scale, the architectural advantage of RadixAttention matters less because the compute is dominated by decode, not prefill. The pattern: SGLang wins big on small models and prefix-heavy workloads (RAG, agents, multi-turn chat). vLLM closes the gap on large models and unique-prompt batch jobs where there’s no shared prefix to cache.

Feature	SGLang	vLLM
Pricing (managed API)	$0.85/M input, $2.55/M output	$1.10/M input, $3.30/M output
Core caching	Token-level radix tree (RadixAttention)	Block-level hash-based (PagedAttention)
Structured generation	Native, scheduler-integrated	External (Outlines)
Best workload	RAG, agents, multi-turn, prefix-heavy	High-throughput batch, unique prompts
Hardware support	NVIDIA, AMD (partial), Moore Threads	NVIDIA, AMD, Intel, TPU, CPU fallback

The latest SGLang release, v0.5.15 on July 10, 2026, shipped zero-overhead Spec V2 speculative decoding (+11% end-to-end TPS), Breakable CUDA Graph as the default capture path, IndexShare MTP for long-context draft optimization, and native Exa web search support. The speculative decoding gains matter most at low batch sizes — exactly the latency-sensitive agent loops where SGLang already excels. If you’re running multi-agent orchestration frameworks that make frequent LLM calls with shared context, SGLang’s prefix caching and spec decoding compound.

The Cost Reality: API vs Self-Hosted at Scale

SGLang’s managed API looks cheaper on paper — $0.85 per million input tokens and $2.55 per million output tokens on the base tier, versus vLLM’s $1.10/M input and $3.30/M output. For input-heavy workloads, that’s roughly 20% savings. But the headline numbers hide the real cost structure.

For a 50M tokens/month workload, self-hosted SGLang on spot GPUs costs approximately $1,200/month, while cloud API costs approximately $2,200/month — about 45% savings. At enterprise scale (1B tokens/month), self-hosted on an 8x A100 cluster runs ~$6,500/month versus ~$8,000/month for cloud API. The gap narrows at scale because cloud providers negotiate volume discounts, but self-hosting still wins on raw compute.

The catch: hidden costs including egress, context window overages, and GPU idle time can add 30-50% to cloud inference bills. Context-window overages are the primary culprit — every 10 extra tokens of output beyond 4K context adds 15-20% overhead due to KV cache recomputation. Egress fees from cross-region data transfer add $0.02-$0.09/GB, which compounds fast at high token volumes. And spot instances, while cutting compute costs by 60-70%, introduce preemption risk: one production deployment reported spot interruptions averaging 8% overhead, higher than the historical 5% baseline.

Reserved instances with a 1-year commitment can reduce SGLang compute costs by 40-62%, which is the sweet spot for predictable, continuous traffic. The decision isn’t “API or self-host” — it’s “which costs can you model accurately, and which will surprise you.”

One scenario we cannot project: a 50-developer SGLang deployment cost cannot be estimated because no per-developer token volume or per-seat licensing cost exists in the research. The data only covers infrastructure costs for specific token volumes. If you’re budgeting for a team, you’ll need to profile actual usage patterns first.

The Hardware Inversion: Why Vendors Now Chase SGLang

The most significant structural shift in the inference landscape isn’t a throughput number — it’s that hardware vendors now depend on SGLang for market access, not the other way around. This inversion is what I call the inference substrate pattern: the framework became the chokepoint, and hardware compatibility with it became table stakes.

SGLang supports AMD ROCm via the ATOM backend, and Moore Threads MUSA was merged into the SGLang mainline in May 2026. AMD published a detailed blog on SGLang-ATOM integration for ROCm-native acceleration in July 2026. Moore Threads submitted 47 PRs to SGLang’s mainline, with 41 merged, completing full-chain support from environment setup to distributed inference. Chinese GPU makers aren’t just writing drivers — they’re contributing to the framework’s core.

Here’s the tension: SGLang advertises universal hardware support across NVIDIA, AMD, and Intel, but all performance-critical optimizations remain NVIDIA-first. The v0.5.15 release notes highlight NVIDIA Blackwell optimizations and Breakable CUDA Graph, while the feature matrix lists AMD support as “Partial.” If you’re buying AMD GPUs expecting parity with NVIDIA on SGLang, you’ll find gaps in the performance-critical paths. The framework runs on AMD hardware — it just doesn’t run as fast, and the newest optimizations land on NVIDIA first.

DFlash speculative decoding, integrated into SGLang by Modal, LMSYS, and Z Lab, achieves greater than 4.3x baseline throughput and 1.5x over native MTP for Qwen 3.5 397B-A17B. That kind of community-driven optimization, landing in SGLang’s mainline and benefiting all users, is the compounding advantage. vLLM has a broader community and comparable raw throughput, but SGLang’s ecosystem momentum — Day-0 model support, universal hardware backends, speculative decoding innovations — is accelerating faster.

When to Pick SGLang vs vLLM: A Decision Framework

Your workload profile, not benchmark headlines, should drive this decision. Here’s the framework I use:

Default to SGLang if:

• Your workload involves multi-turn conversations, RAG pipelines, or agentic loops with shared system prompts
• You need native structured generation (JSON schemas, regex constraints) without external tool overhead
• You’re deploying on multiple hardware platforms and want a framework where vendors are actively contributing backends
• You run latency-sensitive agent loops where speculative decoding at low batch sizes pays off

Default to vLLM if:

• You’re doing high-throughput batch inference with unique prompts (no shared prefixes to cache)
• You need the broadest hardware support including TPU, Intel Gaudi, and CPU fallback
• Your team values operational simplicity — vLLM’s pip-install-and-go model has lower setup friction
• You’ve profiled your workload and found a specific throughput bottleneck that vLLM’s superior GPU utilization solves

The compounding factor: SGLang’s ecosystem advantage — Day-0 model support, vendor backends merging into mainline, speculative decoding innovations — will outweigh marginal throughput differences within 12-18 months. If you’re starting a new production deployment today, SGLang is the safer long-term bet unless you have a profiled, specific reason to choose vLLM. The LangGraph production tradeoffs we’ve analyzed follow a similar pattern: the framework with stronger infrastructure primitives wins over time, even if a rival shows better numbers on isolated benchmarks.

For teams already running OpenAI Agents SDK or similar frameworks, SGLang’s OpenAI-compatible API means you can swap your inference backend without touching application code. That portability is the real unlock — you’re not betting on a framework, you’re betting on an interface that multiple frameworks support.

The Open Question

SGLang’s trajectory raises a question worth sitting with: when a single open-source framework becomes the default substrate for an entire layer of the AI stack, does the competitive landscape shift from “which engine is fastest” to “which engine has the deepest vendor relationships”? The throughput war is converging — vLLM 0.25 and SGLang 0.5.15 both shipped the same week in July 2026, both targeting the same CPU-GPU sync stall, both squeezing the same last few percent of idle time. If raw performance converges, the differentiator becomes ecosystem: who supports your hardware, who ships your model on Day 0, who integrates your speculative decoding research first. SGLang is winning that race. Whether that concentration of power in one framework is healthy for the ecosystem long-term — that’s a question worth asking before you build your entire inference strategy around it.