LLM Serving Stack: Why Per-Token Prices Hide Real Costs

xAI’s Grok 4.1 Fast endpoint has been silently redirecting requests to Grok 4.3 pricing at $1.25/$2.50 per million tokens since May 15, 2026 — more than 6x the original input rate — with zero customer notification and no error thrown. Teams that haven’t audited their model slugs against invoices have been overpaying for two months without a single code change. That’s the real LLM serving stack in mid-2026: the number on the pricing page is the least interesting part of what you pay. Your actual cost is a function of workload shape, serving engine efficiency, routing choices, gateway markups, and hidden slug redirections that most teams never check. The era of optimizing LLM costs by picking the cheapest model is over. The next wave of savings comes from optimizing the full inference stack.

Sticker Prices vs. What You Actually Pay

Closed frontier APIs — GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro — run roughly $2–$5 per 1M input and $9–$30 per 1M output, per TechSy’s pricing comparison. Open-weight models hosted on specialist providers are 5–50x cheaper per token, often priced at $0.05–$0.60 per 1M tokens, according to StackSpend. That gap looks like an obvious win for open models. It isn’t that simple.

Production LLM inference costs include hidden factors beyond sticker prices: system prompts, retrieved context, tool schemas, conversation history, cached prefixes, structured outputs, retries, and reasoning tokens, per DecodeTheFuture. A “cheap” model that requires more retries or generates more verbose outputs can cost more per completed task than a pricier model with better token efficiency. The cost per completed task matters more than the cost per token alone, as DecodeTheFuture’s analysis puts it.

Here’s a concrete example of why this matters. DeepSeek V4 Flash is priced at $0.14/$0.28 per million tokens — 30x cheaper than GPT-5.5’s $5/$30. But if a workload requires multiple retries because the cheaper model produces lower-quality outputs, the total token consumption per successful task can erase that gap. Software Thug notes a similar dynamic: Grok 4.5 is cheaper than GPT-5.4 mini on output, but lower quality leads to more retries and higher total cost.

Output tokens typically cost 3–6x more than input tokens across providers, per TechSy. Verbose responses hurt. Cached input tokens are billed at roughly 10% of the normal input rate on providers that support prompt caching. Every major model provider offers a Batch API at a flat 50% discount on both input and output tokens. These two mechanisms — caching and batching — are where most teams find immediate savings without changing models.

The Same Model Costs Differently Everywhere

The same open model costs different amounts depending on which host serves it. GPT-OSS-120B runs ~$0.15/$0.60 on Groq, ~$0.039/$0.19 on DeepInfra, and ~$0.35/$0.75 on Cerebras, per StackSpend. You’re buying each provider’s hardware, throughput characteristics, and margin — not just the weights.

Provider	GPT-OSS-120B Input/1M	GPT-OSS-120B Output/1M	Target Audience
DeepInfra	$0.039	$0.19	Cost-optimized throughput workloads
Groq	$0.15	$0.60	Latency-sensitive workloads needing fast TTFT
Cerebras	$0.35	$0.75	Max decode speed, premium pricing

Gateway aggregators add another layer. Requesty charges 50% less than OpenRouter for identical flex-tier OpenAI models, as it passes through lower provider tiers, per Requesty’s data. But aggregators also add credit fees on top of token prices, and markup structures vary widely across gateways. For high-volume workloads, direct hosting via providers like DeepInfra or Groq is often cheaper than using an aggregator once markup is accounted for.

The decision isn’t “which model” — it’s “which model on which host through which gateway.” Each layer adds or removes cost, and most teams only look at the first layer. If you’re routing everything through a single gateway without comparing per-model markups, you’re likely leaving money on the table. This is where LLM routing becomes the highest-leverage architectural decision: routing layers that send each request to the best-priced model for the task can cut costs 60–80% without quality loss.

Serving Engines: The Throughput Frontier Moved This Week

In the same week — July 10–11, 2026 — both vLLM and SGLang shipped major throughput wins by removing CPU–GPU synchronization stalls, per Dreaming Press. The frontier of serving throughput in mid-2026 isn’t a faster kernel — it’s the war on the stall, where the accelerator sits idle waiting for the host to hand it the next task.

vLLM v0.25.0, released July 11, made Model Runner V2 the default for all dense models and removed legacy PagedAttention, per GitHub. The async-first core overlaps host and device work — the scheduler prepares step N+1 while the GPU runs step N. Wins are largest at small-to-medium batch sizes, which is what real agent traffic produces.

SGLang v0.5.15, released July 10, introduced zero-overhead Spec V2 speculative decoding with +11% end-to-end TPS and IndexShare MTP with up to 1.9x lower draft-step cost at long context, per GitHub. For low-concurrency, latency-sensitive agent loops where speculative decoding pays, SGLang’s Spec V2 is the one to benchmark.

The tradeoff is workload-dependent. vLLM’s TileRT integration, released July 14, pairs vLLM prefill with TileRT decode for latency-critical workloads through a public connector interface, per the vLLM blog. But the TileRT pairing targets latency-bound workloads, not throughput-bound batch workloads where native vLLM decode is more cost-effective. Teams running high-throughput batch workloads see no cost benefit from TileRT and may even see higher costs from the added layer. If you’re self-hosting, SGLang cuts inference costs compared to cloud APIs.

Hidden Cost Traps Most Teams Never Audit

The biggest hidden cost in production LLM serving isn’t uncached prompts or suboptimal routing — it’s silent model slug redirection. xAI’s Grok 4.1 Fast endpoint has redirected to Grok 4.3 pricing ($1.25/$2.50) since May 15, 2026, with no customer notification, per Awesome Agents. The legacy slug retires on August 15, 2026. Anyone still routing to grok-4-1-fast-reasoning or -non-reasoning in production has been overpaying for two months without a code change. Most teams don’t audit model slugs against invoices — they check the slug they sent, not the model field on the bill.

Peak-hour pricing is another trap. DeepSeek V4 introduced peak-hour pricing of $0.28/$0.56 per 1M tokens for V4 Flash, double the off-peak rate of $0.14/$0.28, effective July 15, 2026, per Awesome Agents. If your workload runs during peak hours and you budgeted on the off-peak rate, your costs just doubled with no announcement. DeepSeek V4 Flash remains the best raw value for workloads that can run off-peak or stay in US timezones.

Gemini 3.6 Flash launched July 21 at $1.50/$7.50 per 1M tokens, consuming 17% fewer output tokens than Gemini 3.5 Flash — up to 65% on DeepSWE, per Google’s blog. That’s a rare case where token efficiency compounds with a price cut. But the 17% figure is index-wide; the 65% figure applies to specific benchmarks. Anyone budgeting on the higher number is budgeting on a best case.

New Models and Day-0 Serving Support

The serving stack’s value increasingly depends on how quickly it supports new architectures. Kimi K3 is a 2.8-trillion-parameter open-weight model with a 1-million-token context window; full weights are scheduled for release on July 27, 2026, per the vLLM blog. vLLM is preparing day-0 serving with KDA-aware prefix caching, multimodal integration, and tool-calling parsers. Inkling, a 975-billion-parameter multimodal model with up to 1M context, gained day-0 support in both SGLang and vLLM as of July 15, 2026, per LMSYS.

Hugging Face integrated vLLM into HF Jobs, enabling one-command deployment of high-throughput LLM serving, per the Silicon Report. This lowers the barrier for smaller teams to achieve inference performance previously reserved for those with significant infrastructure investments. The bottleneck shifts from serving stack complexity to model quality and workflow integration.

At the edge, Acrab’s GΞLIX 1 processor claims a prefill rate of 1,416.8 tokens per second on a Gemma 26B A4B configuration, roughly 7.5x faster than a Mac Mini M4 Pro, per Unite.AI. That’s a vendor number, not an independent test, and covers prefill alone — no decode-speed figure was published. But it signals where the stack is heading: capital expenditure on local hardware replacing recurring per-token fees for privacy-sensitive or latency-critical workloads.

Daum’s AI Overview service runs on approximately 24 chips across three server nodes, processing around 500 million tokens daily, per Aju Press. That’s a real production deployment on domestic AI infrastructure — a data point worth tracking if you’re evaluating non-hyperscale serving paths.

Building Your Cost-Aware Serving Stack

The teams paying the least for LLM inference aren’t the ones with the cheapest model — they’re the ones who treat the serving stack as infrastructure and optimize each layer. Here’s the decision framework I’d use:

1. Audit your invoices against model slugs, not just the slugs you send. The Grok 4.1 Fast redirection proves this is the largest hidden cost source for production teams. Check the model field on your bill monthly.
2. Match serving engine to workload shape. Throughput-bound with many concurrent users? vLLM’s MRv2 default. Low-concurrency, latency-sensitive agent loops? SGLang’s Spec V2. Long-context serving? SGLang’s IndexShare MTP. Don’t pick an engine on reputation — pick it on your traffic pattern.
3. Compare gateway markups per model, not just per gateway. Requesty charges 50% less than OpenRouter for identical flex-tier OpenAI models. But for high-volume workloads, direct hosting may be cheaper than either aggregator. Run the math on your actual traffic.
4. Use batching and caching aggressively. Every major provider offers a Batch API at a flat 50% discount. Cached input tokens bill at roughly 10% of the normal rate. These are free savings that most teams underuse.
5. Measure cost per completed task, not cost per token. A cheap model that retries three times costs more than an expensive model that succeeds once. Track the full workflow cost including system prompts, retrieved context, tool calls, and reasoning tokens.

OpenAI Presence, launched July 22 as an enterprise product for voice and chat agents, uses OpenAI models for the core agent while allowing third-party models for guardrails and tools, per OpenAI’s announcement. That architecture — core model locked, peripheral models swappable — is a preview of how the stack is fragmenting. The question isn’t which single model to standardize on. It’s which AI gateway architecture lets you swap models per task without rewriting your application layer. And if your agent contexts are bloating with noise, context compression cuts costs with minimal accuracy loss — a lever most teams haven’t pulled yet.

The real question for your team: when was the last time you checked the model field on your invoice against the slug you thought you were calling?