Get LLM serving wrong — set it up the naïve, straightforward way — and an expensive GPU capable of trillions of operations per second will sit idle 60–80% of the time. The bottleneck is rarely raw compute; it's scheduling and memory. Closing that gap is the entire job of a modern inference engine, and this article is about the ideas that did it.
To understand why, you need one idea about how these models generate text. An LLM produces output autoregressively: one token at a time, where each new token requires a full forward pass that looks back at every previous token. That "looking back" is expensive — and avoiding it is where our whole story begins.
First, the thing everything else is about: the KV cache
Inside every attention layer, each token is turned into three vectors: a Query, a Key, and a Value. To generate the next token, the model takes the current token's Query and compares it against the Keys of all previous tokens, then blends their Values accordingly. That is attention in one sentence.
Here's the catch. To produce token #101 you need the Keys and Values of tokens #1–100; for token #102, tokens #1–101. Recomputing them from scratch every step would be painfully slow, so the model stores the Key and Value vectors for every token it has already seen and reuses them. That store is the KV cache — a growing list of (Key, Value) pairs, one entry per token.
Figure 1 · During decoding, the KV cache grows by one entry per generated token
Filling that cache reveals something useful: every request runs in two very different phases. Think of it like answering a question. First you read the whole question — you can take it in at a glance, all the words at once. Then you write your answer — necessarily one word after another, each word depending on the ones before. LLMs work exactly the same way.
Why this matters for the rest of the article: decode is the long, unpredictable part. Nobody knows in advance whether a request will decode 5 tokens or 1,500 — and that single fact is the root of nearly every problem ahead. It's why short requests get stuck behind long ones, why memory can't be sized correctly up front, and why keeping the GPU busy is so hard. Almost everything that follows is really about making the decode phase efficient.
Now the connection that ties this whole article together. You want to serve as many users at the same time as possible — not just to keep users happy, but also to make sure your GPU is utilized efficiently. But keep in mind that every request drags along its own KV cache. So a GPU serving lots of people has two things it must never waste: its compute (the slots doing the actual token-generation work) and its memory (the KV caches those slots occupy). The rest of the story is about squeezing more out of both — the first two eras keep the compute busy, and the third finally makes the memory dense.
Act II — ORCA / continuous batching: schedule per token, not per request. Kills idle compute, but memory is still wasteful.
Act III — vLLM / PagedAttention: treat the KV cache like a computer's virtual memory. Waste collapses to near zero.
Additional note — Hardware (Groq): if moving data is the problem, put the whole model on the chip.
Act I — Padding and Static Batching
Serve requests one at a time and the GPU sits badly underused — like a bank with a long line but only one counter open, the other tellers just looking at their phones and taking a break. Batching is the fix: open all the counters at once. Instead of pushing one request through the model and then the next, you run a whole group of them together in a single pass. GPUs are built for exactly this — doing the same operation on many things at once — so serving many requests together is what keeps all those idle tellers working.
Where the intuition comes from: batching images to train a CNN
If you've ever trained a CNN, you know the shape of this problem. You never feed images one at a time; you stack a mini-batch into a single 4-D tensor of shape [batch size, color, height, width] and push it through in one pass. But there's a precondition everyone hits on day one: every image must have the same dimensions. So you resize or pad each one to a common height and width before stacking — compute spent on pixels that carry no real signal.
LLM batching inherits the same constraint, just along the sequence axis. To stack requests into one tensor of shape [batch size, sequence length, hidden size], every sequence must be the same length. Real prompts aren't. So we do the CNN thing: pad the short prompts with filler tokens up to the length of the longest — and the GPU dutifully burns compute on that filler.
Figure 2 · Padding a batch to the longest sequence
Then the real waste begins: waiting for the slowest
Padding the input is only half the problem. The bigger issue is the output. In static batching, the batch is fixed for its whole lifetime: you admit N requests, run them together, and return results only when all of them finish. But output lengths are wildly unpredictable — one user wants a yes/no answer (5 tokens), another a full code review (1,500 tokens).
So the short requests finish early… then just sit there, their slots occupied and doing nothing, until the single longest request grinds to a halt. Your dashboard says "GPU busy, batch of 8." In reality it's been processing a batch of 1 for most of that window.
Figure 3 · Static batching — the whole batch runs at the speed of its slowest member
- Head-of-line blocking. A 5-token request is held hostage by a 1,500-token one in the same batch — its latency set by the slowest neighbor, not its own work.
- Collapsing utilization. As requests finish at different times the batch drains, doing work for fewer and fewer sequences — but can't admit new ones until everyone is done. Mixed workloads land at just 20–40% GPU utilization.
- Padding tax. Compute is spent on filler tokens that carry no information.
The problem is structural, not a tuning knob: the batch is frozen for its whole lifetime. What if we could change its membership while it runs?
Act II — ORCA and Continuous Batching
In 2022, the ORCA paper (Yu et al., OSDI '22) answered that question with a simple reframing. The old world scheduled at the granularity of a request. ORCA schedules at the granularity of an iteration — a single token-generation step. This is iteration-level scheduling, better known today as continuous batching.
The intuition: a bus that never stops moving
Static batching is a tour bus that waits until every seat is full, drives the whole route, and won't let anyone off until the final stop — even the passenger who only needed the first block. Continuous batching is a city bus: at every stop, whoever has arrived gets off and whoever is waiting gets on. Always full, always moving, nobody held up by the longest rider.
Concretely, at every decode step the scheduler:
1. Generates one token for every active request → 2. Evicts any request that just hit its stop token → 3. Pulls waiting requests from the queue into the freed slots → 4. Repeats.
The batch's composition now changes every step. A finished 5-token request returns to the user immediately instead of being trapped behind a 1,500-token neighbor.
Figure 4 · Continuous batching — a freed slot is refilled on the very next step
The catch nobody mentions: you just broke your nice rectangular batch
Continuous batching sounds clean until you try to run one forward pass on it. Recall Act I: batching worked because every sequence was the same length, so they stacked into one neat tensor. But continuous batching deliberately does the opposite — at any step the batch is a ragged mix of lengths:
Request A is 3 tokens into its answer. Request B just got admitted and is doing a 5-token prefill. Request C is 200 tokens deep. Same batch, same forward pass, three different lengths.
You can't pad this back into a rectangle — that revives the padding tax you paid ORCA to remove. So how do you batch a batch that isn't a rectangle? ORCA's second key idea: selective batching.
The insight: some operations don't care about sequence length, and one does
Sort a transformer layer's operations by one question: does this operation mix information across different token positions?
| Operation | Looks across positions? | Batchable? |
|---|---|---|
| Linear / QKV projection | No — same weights applied to each token row independently | ✅ Yes |
| LayerNorm | No — normalizes each token's own vector | ✅ Yes |
| GeLU / MLP | No — pure per-token function | ✅ Yes |
| Residual Add | No — element-wise per token | ✅ Yes |
| Attention | Yes — each token must look at its own sequence's past KV cache | ❌ No |
Almost every operation is token-independent: it applies the same weight matrix to each token, one row at a time, never looking left or right. For those, sequence boundaries don't matter — so you throw away the [request, length] structure, dump all tokens from all requests into one tall matrix of shape [total tokens, hidden size], and run a single big matrix multiply.
Attention is the one exception — it's exactly the operation where a token gathers from every previous token in its own sequence: A's tokens attend to A's cache, not B's or C's. Since each request has a different number of past tokens, their Key/Value tensors have different shapes and can't fold into one batched multiply. So attention — and only attention — is peeled off and run per request.
That's the whole trick. Batch the token-independent operations across the flattened batch; split out only attention, run it per sequence, then merge back. You keep one giant efficient GEMM for the vast majority of the work and pay the per-request cost only where you truly must.
Figure 5 · Selective batching, step by step (batch: A=3, B=5, C=2 tokens)
ORCA showed a 36.9× throughput improvement over previous inference techniques at the same latency — a jump big enough that continuous batching is now the default in essentially every serious serving stack.
ORCA kept the compute full. But it exposed the other resource: memory. To swap requests in and out, the server still had to reserve a block of GPU memory for each request, sized to the longest it might possibly get.
Why the maximum? The server can't know in advance how long a request will run — the model decides when to stop, token by token. So to avoid running out of room mid-generation (which would crash the request), it books the worst case up front. That reservation is dead weight: it's off-limits to everyone else whether or not it's actually filled.
- Internal fragmentation. A request that books room for 2,048 tokens but generates 200 still holds the whole reservation — the other ~1,800 slots sit empty but locked.
- External fragmentation. As variable-length requests come and go, free memory shatters into gaps too small to fit a new reservation — even when the total free memory is plenty.
The measurements were brutal: 60–80% of KV-cache memory wasted. And since each concurrent request needs its own cache, memory is the real cap on how many you can batch. We needed a smarter way to store the cache.
Act III — vLLM and PagedAttention
In 2023, the vLLM project (Kwon et al., SOSP '23) landed the final blow, by borrowing an idea operating systems solved decades ago: virtual memory with paging.
The intuition: stop reserving a whole parking row per car
The old KV cache is a parking lot where every car must reserve a run of spaces as long as the longest car ever recorded, all side-by-side. Compacts waste most of their reservation and the lot fills with awkward gaps. PagedAttention rips out that rule: it chops the lot into small, uniform blocks (16 tokens each) and lets a car's KV cache occupy any free blocks, anywhere, not necessarily adjacent. A small block table (like an index) tracks which blocks belong to which sequence and in what order.
The sequence still sees a clean, continuous run of tokens — the "virtual" view. Physically, its blocks are scattered wherever there was room, handed out on demand one at a time as the sequence grows.
Figure 6 · Same three sequences, same memory pool — reserved vs. paged
Because allocation is now fine-grained and on-demand, vLLM packs far more sequences into the same memory — utilization jumps from ~20–40% to about 96%. A bonus falls out for free: sequences that share a prefix (say, the same system prompt) can point to the same physical blocks instead of duplicating them — the basis of prefix caching.
Additional Note — When Software Isn't Enough: Hardware-Assisted Inference
Notice what every problem so far shared. Padding, idle slots, fragmentation — all symptoms of one physical fact: a GPU keeps the model's data in a large pool of memory that sits next to the chip, not on it. Every step, the chip must fetch weights and the growing KV cache out of that separate memory before it can compute the next token. The fetch is slow relative to the math, so the compute units spend much of their time simply waiting for data. Padding, ORCA, and PagedAttention all make that waiting hurt less. But what if you removed it entirely?
That's the bet behind Groq's LPU (Language Processing Unit), a chip designed specifically for running models, not training them.
Two ideas, one goal
1. Keep the data on the chip. Instead of fetching the model from nearby memory every token, the LPU holds its working set in fast memory built into the chip, so the dominant cost of decoding — waiting for weights — largely disappears. 2. Plan everything ahead of time. Where a GPU decides what to run next on the fly (causing unpredictable slow-downs), the LPU works out the exact schedule in advance, so every response takes a consistent, predictable amount of time.
The payoff is remarkable single-stream speed — public benchmarks put popular models in the hundreds of tokens per second, several times a comparable GPU endpoint, with very little delay before the first word. It's a different goal than vLLM's: vLLM maximizes total throughput across many users on one GPU; the LPU minimizes latency for the single response in front of you.
- Little memory per chip. On-chip memory is small, so a big model won't fit on one chip — you split it across many chips wired together, raising system cost.
- Different sweet spot. Great for low-latency, one-at-a-time serving; it doesn't win the many-users-at-once throughput game a vLLM-packed GPU does.
- Less flexibility. Models must be compiled to the specific hardware, and access is largely cloud-only rather than a card you drop into any server.
Hardware and software aren't rivals here — they're complementary answers to the same problem. Most of the world still serves on GPUs with vLLM; specialized chips like the LPU carve out the latency-critical frontier.
The takeaway
The story of LLM inference engines isn't really about faster math. Every era ran the same transformer. What changed was an insight about the two scarce resources — the chip's compute and its memory — and, in the end, about the cost of moving data.
Padding taught us to batch, then trapped us behind the slowest request. ORCA freed the scheduler to think per-token, and used selective batching to keep the math efficient on ragged batches — but ran into the memory wall. vLLM borrowed a decades-old operating-systems trick and turned the KV cache into flexible, on-demand memory. And Groq asked whether we should keep working around the cost of moving data, or just build a chip that keeps the data close. The lesson worth keeping: when the hardware isn't the bottleneck, the win is in how you schedule work and lay out memory — and when even that isn't enough, you change the hardware.
Next frontiers on this same foundation: Chunked prefill Prefix caching Speculative decoding Disaggregated prefill/decode SGLang TPUs / Cerebras
Sources & further reading
- Mishra, The Evolution of LLM Inference Engines: How vLLM Changed the Game (Medium) — the article that started this deeper dive.
- Yu et al., ORCA: A Distributed Serving System for Transformer-Based Generative Models, USENIX OSDI '22 — iteration-level scheduling & selective batching (36.9× throughput at equal latency).
- Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention, SOSP '23 — the vLLM paper (60–80% fragmentation → ~96% utilization).
- Bamania, 5 LLM inference batching techniques every AI engineer should know (Into AI) — further reading on prefill/decode, chunked prefill, and prefill–decode disaggregation.
- Groq — LPU architecture: model kept in fast on-chip memory, schedule planned ahead of time for predictable, low latency.