Continuous Batching: The GPU Trick With a Hidden Ceiling

Orca’s iteration-level scheduling paper reported a 36.9x throughput improvement over NVIDIA FasterTransformer at equal latency, and that single result reshaped how every inference engine schedules work. Continuous batching — the technique Orca introduced — lets an LLM serving system swap requests in and out of a running batch at every decode step instead of waiting for the slowest request to finish. It’s the reason modern GPUs can stay near full utilization under uneven traffic. It’s also where a less obvious performance ceiling begins.

The throughput gains are real and well-documented. The production reality is more complicated. When you dig into measured latency profiles, cluster utilization data, and the CPU-GPU coordination gaps that synchronous batching leaves open, you find that continuous batching solves one bottleneck while leaving two others largely untouched. Understanding those limits matters more than celebrating the headline number, especially if you’re making build-versus-buy decisions for inference infrastructure.

How Does Continuous Batching Actually Work?

Continuous batching schedules at the iteration level, not the request level. When a sequence finishes generating, its slot is filled by a waiting request on the very next token step — not on the next batch boundary. This eliminates the head-of-line blocking that makes static batching inefficient under variable-length traffic, per packet.ai’s explanation.

Here’s why that matters. Under static batching, the server collects a fixed number of requests, runs them all together, and waits for every single one to finish before starting the next batch. If one request generates 3 tokens and another generates 2,000, the short request’s slot sits dead for 1,997 steps. Static batching throughput can fall as low as 81 tokens per second under high sequence-length variance because every request in a batch waits for the slowest one to finish, according to packet.ai’s analysis.

Continuous batching drops the fixed batch boundary entirely. The scheduler re-decides batch membership at every decode step. A finished sequence is evicted immediately, a waiting request is admitted, and the GPU never idles waiting for a batch to drain. The technique traces to Orca (OSDI 2022), which introduced iteration-level scheduling and reported that 36.9x throughput improvement — a figure confirmed across multiple independent sources, per packet.ai.

Every major inference engine implements the same core idea under a different name. vLLM and TGI call it continuous batching. TensorRT-LLM calls it in-flight batching. LMDeploy calls it persistent batching. SGLang uses the continuous label too. They all share the same iteration-level scheduling foundation, as packet.ai’s analysis documents.

One distinction worth noting: continuous batching is a scheduling technique, distinct from PagedAttention, which is a memory management technique. The two are complementary, not the same thing. PagedAttention reduces KV cache fragmentation and makes the memory stretch further, which makes continuous batching more effective — but you can have one without the other.

What’s the Throughput Improvement in Practice?

The benchmark numbers are striking. Anyscale measured up to 23x throughput improvement with continuous batching over static batching using vLLM, a figure repeated across independent analyses, per DigitalOcean. Morph runs continuous batching in production for morph-v3-fast at approximately 10,500 tokens per second, according to Morph.

Technique	Throughput	Key Tradeoff	Target Scenario
Static batching	As low as 81 tok/s under variance	Head-of-line blocking; dead slots	Uniform workloads, batch jobs
Continuous batching	Up to 36.9x improvement (Orca)	P99 jitter from prefill/decode mixing	Live traffic, multi-user serving
Async continuous batching	Reduces ~25% CPU-GPU idle	Implementation complexity	High-throughput production serving

Those are saturation numbers, though. They describe what happens when the GPU has enough concurrent requests to fill every slot and keep the arithmetic units busy. Production traffic rarely looks like that — and that gap between benchmark throughput and realized throughput is where the story gets interesting.

Does Continuous Batching Actually Improve Latency?

Here’s where the narrative gets nuanced. Continuous batching improves median (P50) latency because requests are admitted near-instantly instead of waiting for a batch to fill. But it can worsen P99 tail latency by introducing streaming jitter during token generation, per DigitalOcean’s measured testing on an H200 GPU with vLLM v0.24.0 and Llama 3.1 8B.

The mechanism is straightforward. With static batching, most delay occurs at admission — you wait for the batch to fill, then you wait for the slowest request. Once your request starts, it runs uninterrupted. Continuous batching eliminates the admission wait but introduces occasional pauses during streaming because new requests entering the batch compete for GPU time with ongoing decodes. The latency variance doesn’t disappear — it redistributes.

There’s also a real tradeoff from mixing prefill and decode work in the same batch. Prefill is compute-heavy — the model processes the entire prompt in one forward pass. Decode is memory-bound — each new token requires its own forward pass. When you mix them, a new request’s prefill can stall ongoing decodes. Chunked prefill, built into vLLM’s V1 engine by default, is the technique most engines use to manage this, per packet.ai. It breaks prefill into smaller chunks that interleave with decode steps, reducing the jitter but not eliminating it.

If you’re serving a chat application where users perceive streaming speed as quality, P99 jitter matters more than P50 improvement. A request that starts fast but stutters mid-stream feels worse than one that starts slightly slower but streams smoothly. This is the tradeoff that most “continuous batching universally improves latency” summaries skip.

What Happens to GPU Utilization at Cluster Scale?

This is where the throughput story runs into production reality. Cast AI’s 2026 State of Kubernetes Optimization Report, drawn from roughly 23,000 production clusters across AWS, Azure, and GCP, measured average enterprise GPU utilization at 5 percent, per Forte Group’s build-vs-buy analysis.

Five percent. That number should reframe every conversation about inference optimization. Continuous batching is designed to keep a GPU busy when there’s work to do. It does nothing to create work. If your traffic is bursty — concentrated during business hours, near-zero overnight — your GPU sits idle regardless of how efficiently it batches when it’s active.

The cost math breaks down like this. Spheron’s 2026 analysis shows an eight-GPU H100 SXM5 pod at roughly $19.20/hr serving Llama 3.1 70B in FP16 through vLLM at 2,800 tok/s yields approximately $1.90 per million tokens, per Forte Group. That’s a competitive number. It’s also a ceiling, not an expectation — because it assumes the pod is saturated. The same analysis models a realistic internal deployment with 500 daily active users, traffic concentrated between 9 and 6 on weekdays, near zero overnight and on weekends. Under that traffic pattern, the $1.90/M token figure balloons because the denominator (realized throughput) collapses while the numerator (hourly cost) stays fixed.

This is the pattern I call the coordination ceiling: continuous batching eliminates discrete batch boundaries and solves token-step packing, but it may leave cluster-scale traffic sparsity and CPU-GPU turn-taking unaddressed, exposing coordination ceilings. The throughput win is real only at saturation, which production rarely hits. If you’re evaluating whether to self-host inference, instrument your workloads for a full quarter before buying anything — the gap between theoretical and realized GPU utilization is where build-vs-buy spreadsheets break.

What’s the CPU-GPU Idle Gap Nobody Talks About?

Even when continuous batching is working correctly and the GPU has a full batch, there’s a second source of waste that most discussions skip. Synchronous continuous batching leaves the CPU and GPU idle in turn-taking gaps that can account for nearly a quarter (≈25%) of total runtime, per Hugging Face’s async batching analysis.

Here’s why. In a synchronous batching loop, the CPU prepares the next batch — selecting requests, updating the KV cache table, evicting finished sequences, admitting new ones — while the GPU sits idle. Then the GPU runs its forward pass while the CPU sits idle. In a loop running hundreds of steps per second, those idle gaps accumulate into real throughput loss. The CPU and GPU are never doing useful work at the same time.

Asynchronous batching fixes this by disentangling CPU batch preparation from GPU batch compute. The CPU prepares the next batch while the GPU computes the current one, so both run in parallel and the GPU stays productive. This is a coordination problem, not a batching problem — and it’s the kind of gap that continuous batching alone doesn’t address.

The data suggests that continuous batching’s throughput wins may plateau in production precisely because the technique solves token-step packing while leaving these coordination gaps open. The next performance leap in LLM serving likely won’t come from better batching — that’s largely been solved. It’ll come from closing the gaps between components that are supposed to work together but spend a quarter of their time waiting for each other.

How Does This Compare to Continuous Manufacturing in Pharma?

The parallel is instructive. Pharmaceutical manufacturing has been moving from batch production toward continuous manufacturing — where materials flow through integrated unit operations without stopping between steps. The global market for continuous biomanufacturing reached an estimated $218 million in 2023 and is projected to reach $599 million by 2028, per Bioprocess International. Continuous, automated production has delivered operating cost reductions of up to 30% for some manufacturers, per Innovatrix.

Yet most plants aren’t truly continuous. Many pharmaceutical facilities already operate continuous unit operations — milling, compression, coating — but the overall system remains batch because material stops, waits, and is stored between steps. As Contract Pharma puts it, much of today’s manufacturing is best described as “batch manufacturing with continuous islands” rather than true continuous manufacturing, per their analysis.

The parallel to LLM inference is exact. Continuous batching solves the discrete batch boundary within a single GPU’s scheduling loop — the equivalent of making one unit operation continuous. But cluster-scale traffic sparsity (5% average GPU utilization) and CPU-GPU coordination gaps (~25% idle) are the equivalent of material stopping between steps. You’ve optimized one stage of the pipeline and left the inter-stage coordination unresolved. The ICH Q13 regulatory framework, finalized in 2022, provides a harmonized international standard for continuous pharmaceutical manufacturing, per Assyro’s guide — but regulatory encouragement hasn’t eliminated the organizational and capital barriers to true end-to-end continuity.

A European consortium recently validated a continuous manufacturing platform for pharmaceutical APIs, demonstrating continuous-flow production of fentanyl with AI-monitored real-time analytics, per Manufacturing Chemist. The PIPAC consortium — De Dietrich, Alysophil, Bruker, and Novalix — showed that integrated continuous-flow chemistry with real-time process control can operate under challenging conditions, per Novalix. But the capital outlay, PAT expertise, and operator retraining required for integration are the same barriers that keep most pharma plants in batch-with-islands mode.

When Should You Care About These Limits?

If you’re running a single-GPU inference server with steady traffic, continuous batching’s throughput gains are unambiguous. Enable it, tune your batch size, and move on. The P99 jitter is manageable for most workloads, and chunked prefill in vLLM’s V1 engine handles the prefill/decode mixing reasonably well by default.

If you’re operating a multi-GPU cluster or making a build-vs-buy decision, the coordination ceiling changes your math. The $1.90/M token figure assumes saturation. Real traffic leaves cards near-idle off-peak, and the 5% average utilization measured across 23,000 clusters means most owned GPU capacity is wasted compute you’re paying for. Before adding nodes, fix batching and quantization — but also fix traffic routing. Cache-aware load balancing that matches requests to replicas holding relevant cached prefixes can cut Time to First Token latency dramatically, and feedback-aware model routing can reduce costs by adapting to real traffic patterns rather than sending everything to your most expensive tier.

The open question for inference infrastructure in 2026 isn’t whether to use continuous batching — every serious engine already does. It’s whether the next meaningful performance gain comes from better scheduling within a GPU or from better coordination across the cluster. The evidence points to the latter. Asynchronous CPU/GPU overlap, cache-aware routing, and traffic-aware model selection are where the remaining headroom lives. Continuous batching has already plateaued in practice — and the teams that recognize that will spend their optimization budget where it actually moves the needle.