AIHow AI Works

Inference Optimization

How a served LLM is made cheap: continuous batching, the prefill/decode split, and the throughput-versus-latency tradeoff behind every API price.

Definition

The same model, on the same GPUs, running the same code, can cost 47x more to serve depending on
one number that no user ever sees.
That number is the batch size — how many people’s requests the
server pushes through each forward pass — and inference optimization is, first and mostly, the craft
of keeping it high without making the product feel slow.

Everything else follows from a single awkward fact: throughput and latency are the same dial
turned in opposite directions.
Raise the batch and the server produces far more tokens per second
per GPU, so the cost of a token collapses; every individual user also waits longer between words. No
configuration maximises both. An operator picks a point on that curve, and that choice is what you
feel when a chatbot streams smoothly, and what you pay when the invoice arrives.

How It Works

Two clocks, two bottlenecks

Serving a request has two phases, and the metrics that matter map one-to-one onto them
(Databricks
names them the way the industry now does):

  • Time to first token (TTFT) — how long before anything appears. It covers queueing plus
    prefill: the model reading your whole prompt in one parallel pass. That pass is a dense matrix
    multiplication and it is compute-bound.
  • Time per output token (TPOT) — how fast the words then arrive. Each one is a decode step,
    which reads all of the model’s weights and all of the cached context to produce a single token.
    It is memory-bound.

Two phases, opposite limits — the memory wall page derives why. What
matters here is the consequence: an optimisation that helps one clock usually hurts the other,
and a serving stack is a negotiated settlement between them.

Static batching stalls at the speed of its slowest sequence

The obvious way to batch is to collect eight requests, run them together, and return the results.
It fails, and the arithmetic shows how badly.

Say seven of the eight requests want 100 tokens and one wants 1,000 — a perfectly ordinary spread,
since nobody knows in advance how long an answer will be. With static batching the batch slot is
held until the last sequence finishes. After step 100 seven slots are dead, and the GPU spends the
next 900 steps doing full-width forward passes for one real user. Useful work as a fraction of what
the batch occupied: (7 x 100 + 1,000) / (8 x 1,000) = 21%. Four fifths of the machine is
computing padding.

Continuous batching: schedule the iteration, not the request

The fix is to stop treating a request as the unit of scheduling. Continuous batching — also
called in-flight batching, and introduced as iteration-level scheduling by
Orca (OSDI 2022) — reschedules after
every forward pass. A sequence that emitted its end-of-text token is evicted immediately, a
request waiting in the queue takes its slot on the very next step, and the batch becomes a rolling
population rather than a fixed cohort. Nothing waits for a straggler, because there is no cohort for
a straggler to hold up. Orca reported up to 36.9x the throughput of FasterTransformer at the same
latency
— not from faster maths, but from never running a padded slot.

Admitting a newcomer, though, means prefilling its prompt, and an 8,000-token prefill wedged into a
decode step makes every user in the batch wait for it. That stutter is why
Sarathi-Serve (OSDI 2024) splits a prefill into chunks and
mixes them into decode batches, keeping each iteration roughly the same size — worth 2.6x the
serving capacity
of vLLM on Mistral-7B, and 3.5x under a strict 100 ms latency target. The
name of that paper is “Taming Throughput-Latency Tradeoff”, which is the honest description of the
whole field.

The worked example: one dial, 47x the price

Take a standard deployment: Llama 3.1 70B in FP8 on one eight-GPU H100 SXM node. Three inputs,
all sourced:

  • Weights: 70.6 GB (70.6B parameters at one byte each — see
    quantization).
  • KV cache: 320 KiB per token (KV cache derives it), so a 4,096-token
    conversation carries 1.34 GB.
  • Hardware: eight H100 SXM, each 80 GB of HBM at 3.35 TB/s
    (NVIDIA) — 640 GB and 26.8 TB/s in
    aggregate. Rented at Lambda’s on-demand rate of $3.99 per GPU-hour
    (Lambda), the node costs $31.92/hour.

The two prices below — the rental rate and the API rate quoted against it — are list prices at the
time of writing, and they will move. Read the table for its ratios, which are set by the
hardware and the model rather than by anyone’s price list: the cost per token falls roughly with
the batch size until the caches overtake the weights, and it stops falling after that. Redo the
arithmetic with today’s rates and the shape is unchanged.

A decode step reads the weights once and every batched sequence’s cache once, so with batch size
B:

bytes read per step = 70.6 GB + B x 1.34 GB
time per step       = bytes / 26,800 GB/s        <- this is TPOT
node throughput     = B / time per step
Batch Bytes read per step TPOT Speed one user sees Node throughput Cost per 1M tokens
1 71.9 GB 2.7 ms 373 tok/s 373 tok/s $23.80
8 81.3 GB 3.0 ms 330 tok/s 2,636 tok/s $3.36
32 113.5 GB 4.2 ms 236 tok/s 7,553 tok/s $1.17
128 242.4 GB 9.0 ms 111 tok/s 14,152 tok/s $0.63
384 586.0 GB 21.9 ms 46 tok/s 17,562 tok/s $0.51

Read the two bold numbers first. Serving one user at a time on hardware that costs $32 an hour puts
the price of a million tokens at $23.80 — while the same node, same model, batch of 384, serves
them for 51 cents. That is the entire gap between a hobby deployment and a business. You can
read a provider’s batch size straight off the table: Together AI lists a 70B-class model at
$1.04 per million tokens (pricing) — below what the
batch-of-32 row costs to produce, which means nobody selling at that price is running batches of 32.

And look what the user paid for it. TPOT went from 2.7 ms to 21.9 ms — from 373 tokens per second to
46, an 8x slowdown in the speed of the streaming text — to buy a 47x cut in cost.

Where the free lunch ends

The table’s last two rows are the interesting ones: throughput rises only 24% from batch 128 to
batch 384, while TPOT gets 2.4x worse. Batching stops paying, and the formula says exactly when.

Batching is free while it amortises a fixed cost: the 70.6 GB of weights get read once and serve
the whole batch. But each sequence brings its own cache, and that is a variable cost — bytes that
must be re-read on every step for every user. The two are equal when B x 1.34 GB = 70.6 GB, at
B ≈ 53. Below that, doubling the batch nearly doubles throughput. Above it, the cache reads
dominate, throughput asymptotes, and all you are still buying is latency for your users. (At batch
384 the node is also nearly out of memory — 586 GB of a 640 GB budget — so it could not go much
further even if it wanted to.)

The model holds only while decode stays memory-bound, and it does: at batch 384 the arithmetic runs
at roughly 310 TFLOP/s per GPU against the ~1,979 dense FP8 TFLOPS an H100 offers. It ignores
attention kernels, sampling and the interconnect, so these are ceilings — a real stack lands well
under them, and it is the shape that transfers, not the absolute numbers.

Prefill and decode want different machines

Because prefill is compute-bound and decode is memory-bound, running both on one GPU means every
prompt you admit steals time from every answer you are streaming. Disaggregation takes the
obvious step: two pools of hardware, one sized for compute and one for bandwidth, with the KV cache
shipped across the network between them.
DistServe (OSDI 2024) showed
this serves 7.4x more requests, or meets a 12.6x tighter latency target, than co-located
serving. The two phases were never the same workload, and the last thing you should do is average
them.

Real-World Applications

DeepSeek published its meter reading, and it is this page in production. Its
V3/R1 inference system overview
describes a disaggregated fleet — prefill units spanning 4 nodes, decode units spanning 18 —
averaging 226.75 H800 nodes over a day, at roughly 73.7k input tokens/s per node in prefill
against 14.8k output tokens/s per node in decode. That ratio is the compute-bound /
memory-bound split with a number on it: a node emits output about 5x slower than it ingests
input. The economics that fall out: at an assumed $2 per GPU-hour, $87,072 of compute per
day
against $562,027 of tokens at list price — a theoretical 545% margin — while each user
still saw a comfortable 20–22 tokens per second. That is the curve, with someone standing on a
well-chosen point of it.

A batch API is this tradeoff sold as a product. Anthropic’s
Message Batches API charges 50% of the
standard price
for requests it may take up to 24 hours to answer, and OpenAI’s Batch API does the
same. Nothing about the model changed. You handed the scheduler a 24-hour latency budget instead of
a two-second one, it packed you into whatever batch had room whenever the fleet was quiet, and it
split the savings with you.

Running a model on your own GPU is the worst row of the table. Batch size 1: nobody to amortise
the weight read with, and an effective cost per token an order of magnitude above what an API
charges. Local inference is worth doing for privacy, offline use or control —
essentially never to save money, and the reason is arithmetic, not the quality of your software.

Key Concepts

Batching is what this page owns. The other levers are real, each has its own page, and each is one
sentence here on purpose:

  • Quantization shrinks the fixed term in the formula above — fewer
    bytes per weight means a faster decode step at every batch size, and the memory it frees becomes
    more batch.
  • KV cache management attacks the variable term: fewer KV heads, fewer
    bits, and paged allocation all mean more sequences fit in the same memory, which is the only way
    to reach the big batches at all.
  • Speculative decoding buys single-user latency by spending
    idle compute — which is exactly the resource a full batch no longer has, so it helps least
    where throughput is already best.
  • Mixture-of-Experts cuts the bytes read per token by activating
    a fraction of the parameters, shrinking the fixed term far more aggressively than quantization
    can.
  • Prefix caching skips prefill entirely for a repeated prompt, which is a TTFT optimisation and
    does nothing at all for TPOT.

Challenges

Optimise only for throughput and you ship a product that feels broken. The failure is rarely in
the average — it is in the tail. A scheduler chasing GPU utilisation will admit a large batch and a
long prefill at the moment your user is mid-sentence, and their stream stalls for hundreds of
milliseconds. The dashboard shows tokens per second climbing; the user sees the text stop, and
leaves. This is why serious systems optimise goodput — requests served within a stated latency
target
— rather than raw tokens per second, and why a mean TPOT is nearly useless next to its 99th
percentile.

Optimise only for latency and you cannot price your product. The batch-of-8 row is a lovely user
experience and a $3.36-per-million-token cost base in a market charging $1.04. No engineering closes
that gap, because it is not an engineering gap; it is the batch size. Teams find this out after
launch, working out why the gross margin is negative and discovering that the answer is a scheduler
setting.

The benchmark you ran does not resemble your traffic. Most published serving benchmarks use a
fixed batch and fixed sequence lengths. Real traffic arrives in bursts, with output lengths that vary
by orders of magnitude — and a system tuned on the synthetic load can be worse under the real one,
because it was optimised for a batch that never has to evict anyone.

Long context quietly deletes your batch. The variable term scales with context length as well as
batch size, so the node that held 384 sequences of 4,096 tokens holds 13 at 128k — and
throughput falls roughly 30x with it. Nothing in your configuration changed; the users just
started pasting in documents. The context window is a cost parameter,
not only a capability.

The shift already underway is from throughput to goodput: schedulers that take a latency target
as an input and maximise the number of requests meeting it, rather than tokens produced per second.
That reframing is what makes disaggregation, chunked prefill and SLO-aware admission control one
idea rather than three tricks — each is a way of spending a latency budget the operator actually
declared.

The second is that the workload is changing shape underneath the optimisation. Reasoning models and
agents emit enormous numbers of tokens no human ever reads — chains of thought, tool calls,
retries — which are pure decode, and therefore pure memory-bandwidth cost. As
test-time compute becomes the way models get smarter, decode stops
being the tail of a request and becomes the entire bill. A workload that is mostly output tokens
with nobody waiting on them is also, conveniently, one where the operator can turn the batch dial
all the way up. The tradeoff does not disappear. It gets easier to resolve, because the impatient
human at the other end of it has left the loop.

Common questions

What is inference optimization?

It is everything a serving system does to get more tokens per second per dollar out of a model whose weights nobody is changing. The single biggest lever is batching — running many users' requests through the same forward pass — and the price of pulling that lever is that each individual user waits longer for each word.

What is continuous batching?

Scheduling at the granularity of one forward pass instead of one request. After every step the server drops sequences that have finished and admits waiting ones in their place, so the batch is a rolling set rather than a fixed group. It is the difference between a GPU running at a fraction of capacity and one that stays full.

What is the difference between TTFT and TPOT?

Time to first token is how long you wait before anything appears; it is set by prompt processing, which is compute-bound. Time per output token is how fast the words then arrive; it is set by generation, which is memory-bound. They have different bottlenecks, they respond to different fixes, and a system tuned for one can be bad at the other.

Does a bigger batch always mean more throughput?

No. Batching is free only while it amortises a fixed cost — reading the model's weights. Once the batch's key-value caches are bigger than the weights, every extra request adds bytes that must be read on every step, and throughput flattens while per-user latency keeps getting worse.

Why is a batch API 50% cheaper than a real-time one?

Because you are selling the provider your latency. Given a 24-hour window instead of a two-second one, the scheduler can pack requests into large batches and run them when its fleet is otherwise idle, which raises tokens per second per GPU. The discount is the throughput-latency tradeoff, priced.

What is prefill/decode disaggregation?

Running the two phases of a request on separate pools of hardware, because they hit opposite bottlenecks. Prompt processing wants compute; token generation wants memory bandwidth. Keeping them on the same GPU means each phase interferes with the other's latency, so large serving stacks split them and ship the cache between them.