LLM inference: how it works

LLM inference is what happens every time a trained model answers a request. The serving engine runs the network forward, again and again, until the response is complete. Nothing is learned. No weights change. Training happens once per model and inference happens on every call, so inference is where your users wait and where your bill accrues.

You can't make inference fast in a single dimension. You trade latency for one user against throughput across all users, and both against cost per token. This article walks one request through the engine, explains why its two phases have different bottlenecks, and then covers metrics, batching, the main optimizations and cost. For what happens inside the forward pass itself, see how large language models work.

How LLM inference works: the request lifecycle

Six steps cover every text-generation request:

• Tokenize: The server converts your prompt into token IDs with the model's tokenizer.
• Prefill: The model processes all prompt tokens in one forward pass and stores the attention keys and values it computes.
• Sample the first token: The last position's output is a vector of logits over the vocabulary. Temperature rescales them, softmax turns them into a distribution, and top-p decides how much of that distribution the sampler draws from.
• Decode loop: The new token is appended, the model runs a forward pass for that single position, and the sampler picks again. Every output token costs one pass.
• Stop: The loop ends when the model emits its end-of-sequence token, matches a stop string, or reaches max_tokens.
• Detokenize and stream: Token IDs are converted back to text and sent as they are produced, which is why streaming adds no work for the model.

Steps 1 and 6 are cheap CPU work. Steps 2 and 4 take the GPU time, and they behave so differently that serving systems treat them as separate workloads. The guide to tokens and attention covers what happens inside each pass. This article stays with the engine.

Prefill is compute-bound, decode is memory-bound

Send a 4,000-token prompt and ask for a 500-token answer. The GPU does one large pass over 4,000 positions, then 499 small passes over one position each. Both use the same weights, but they stress different parts of the hardware.

In prefill, every prompt position goes through the model at once, so each layer is a large matrix-matrix multiplication. That is the work GPUs are built for, and it keeps the arithmetic units saturated, which means prefill is compute-bound. Its cost grows with prompt length, so prompt length mostly decides how long you wait for the first token. Faster attention kernels like FlashAttention-3 pay off here, because prefill runs attention over the whole prompt in one go.

A decode step computes one position per request, so each layer is a matrix-vector multiplication, one position against the full weight matrix. That is very little arithmetic, but the GPU still has to read every active weight and the request's cached attention state out of memory. The arithmetic units finish early and wait for data, so decode is memory-bound. Memory bandwidth sets its speed. Together AI's QTIP post on memory-bound decoding and weight quantization gives the upper bound for small-batch decoding: tokens/sec ≤ batch size × memory bandwidth (GB/s) ÷ model size (GB).

Run the numbers for a single stream. Llama 3.1 8B at bf16 is ~16 GB of weights, and NVIDIA's spec sheet gives the H100 80GB 3.35 TB/s of memory bandwidth, so the ceiling is 3,350 ÷ 16 ≈ 210 tokens/sec. Llama 3.1 70B at bf16 needs ~141 GB for weights alone, more than one 80 GB GPU holds. At fp8 it is ~71 GB, which fits with little room to spare, and the ceiling becomes 3,350 ÷ 71 ≈ 47 tokens/sec. These are ceilings, not forecasts, because cache reads and kernel overhead come out of the same budget. The bound also shows what helps decode: fewer bytes per weight (quantization), fewer weights read per token (a mixture-of-experts model reads only its active parameters), and more tokens per read (batching).

That cached attention state is the KV cache: the keys and values for past tokens, stored so decode doesn't recompute them at every step. It grows with every token of every active request, and that memory often caps concurrency before the weights do.

The two phases also compete for the same GPU. When a long prompt arrives while other requests are mid-decode, a naive scheduler runs the whole prefill first, and every streaming response stalls until it finishes. Batching has to manage that conflict.

LLM inference metrics that matter

Four metrics describe an inference deployment, and each one maps back to a phase:

• Time to first token (TTFT): The wait between sending a request and receiving the first token. It is mostly prefill, plus queueing, network time and any cold start.
• Time per output token (TPOT), also called inter-token latency (ITL): The gap between streamed tokens, which is mostly decode. Many dashboards report the inverse, tokens per second (TPS). For a single stream, TPS ≈ 1 ÷ TPOT, so 20 ms per token is 50 TPS.
• End-to-end latency: E2E ≈ TTFT + TPOT × (output tokens − 1). With a 0.5 s TTFT and a 20 ms TPOT, a 300-token chat reply takes 0.5 + 0.02 × 299 ≈ 6.5 s. A reasoning model that emits 4,000 tokens takes 0.5 + 0.02 × 3,999 ≈ 80.5 s, and TTFT is under 1% of it. Long-output workloads are decode-dominated, so work on TPOT first.
• Throughput and goodput: Throughput is total tokens/sec per GPU across all requests, and it sets your cost per token. The share of that throughput delivered inside your latency target is goodput. Size capacity on goodput, because a late token did not serve the user.

Report P50 and P95 or P99 for each of these. Averages hide the tail. Queueing gives latency a long tail, and the tail is what users notice.

To measure them yourself, stream the response and timestamp the first chunk for TTFT. Count output tokens from the API's usage field or the model's own tokenizer. Character and word counts will not match. Then repeat at your production concurrency, because a single request only measures an idle server. The docs page TTFT and TPS explained lists what moves each number.

Batching trades latency for throughput

Serving one request at a time wastes most of a GPU, because decode reads every weight to produce one token. The key idea is simple: read the weights once and use them for many requests. A decode step for 32 requests reads the same weights as a step for one, so throughput rises almost for free until KV cache memory or compute runs out.

Collecting a fixed group of requests and running it to completion is static batching. Short requests wait on the longest one, and new arrivals wait for the whole batch. Scheduling at every decode step instead lets finished sequences leave and queued ones join immediately, and that is continuous batching. Orca (Yu et al., 2022) introduced it as iteration-level scheduling, and vLLM (Kwon et al., 2023) added paged KV cache memory so more sequences fit in each batch.

Batching is not free for the individual request. Once a step's compute and KV cache reads outweigh the weight read, every added request slows the step, so per-request TPOT rises while tokens/sec per GPU keeps improving. The curve has a flat part and then a knee. One published sweep of dedicated endpoints (measured throughput and latency across batch sizes) shows the flat part. It ran Qwen3.5-9B in bf16 on one single-GPU replica (TP1) of the post's demo endpoint, with 200-token generations:

Throughput grew 4× while each request's share stayed flat, and the post reports p95 TTFT rising only ~17% from concurrency 4 to 16. A second profile in the same sweep peaked at concurrency 4 and lost half its throughput beyond it. Find the knee for your model before you set a batch limit.

Batching also brings back the prefill-decode conflict. Splitting a long prompt into pieces that run alongside ongoing decode steps is chunked prefill. Streams slow slightly instead of stalling. Running the two phases on separate GPU pools goes further and is called disaggregation. A production example is cache-aware prefill-decode disaggregation, which also routes each prompt by how much cached context it can reuse.

Inference optimization techniques

Sort every technique by two questions: which phase does it speed up, and does it change the model's outputs. Five techniques cover most of the gains:

• Speculative decoding (exact): A small draft model proposes several tokens and the target model verifies them in one forward pass, keeping only the tokens it accepts, which leaves the output distribution unchanged. Together's docs put the gain at 2–3× TPS when the guesses are right, and it usually shrinks at large batch sizes with short contexts, where less idle compute is left for verification. Adaptive speculative decoding (ATLAS) keeps the draft aligned by learning from live traffic.
• Quantization (lossy): Storing weights at fp8 or 4-bit instead of 16-bit cuts the bytes read per token to a half or a quarter, which speeds up decode and frees memory for the KV cache. fp8 is usually near-lossless and 4-bit varies by model and task, so evaluate LLM quality on your own data first. The docs cover quantization formats and tradeoffs.
• Attention and KV cache optimizations (exact): FlashAttention kernels, PagedAttention and prefix caching preserve outputs while cutting memory traffic, wasted memory and repeated prefill work. Architectures with GQA or MLA go further and shrink the KV cache by design.
• Parallelism (exact): Tensor parallelism splits each layer across GPUs, which lowers per-GPU memory and per-token latency but adds communication at every layer. Expert parallelism spreads a mixture-of-experts model's experts across GPUs instead.
• A smaller model (different outputs): An 8B model reads ~9× fewer bytes per token than a 70B model at the same precision. It is often the cheapest optimization available, provided it passes the same evals.

What LLM inference costs

You pay for inference in one of two units. Serverless APIs charge per million tokens, with separate input and output rates. Output is typically priced at 3–4× input, because output tokens come from decode, the slow, bandwidth-bound phase, while input tokens go through one parallel prefill pass. Dedicated GPUs charge per GPU-hour whether or not you send traffic, so cost per token is the hourly rate divided by the tokens you actually serve.

Together's deployment docs work through an illustrative example. It is not a price quote. An H100 at ~\$3 per hour costs 3 × 24 = \$72 per day. At a serverless rate of \$1 per 1M tokens, \$72 buys 72M tokens, so the crossover sits at ~72M tokens per day. That is 3M tokens per hour, or ~830 tokens/sec sustained around the clock. Below it, per-token pricing is cheaper. Above it, the dedicated GPU wins, provided it can sustain that rate for your model at acceptable latency, which is the batching curve again.

The docs guide to choosing between serverless, dedicated, and containers covers the full decision.

Conclusion

LLM inference is two workloads sharing one GPU. Prefill is compute-bound and sets TTFT, decode is memory-bound and sets TPS, and batching decides where you sit between per-user latency and cost per token. The quickest way to see this is to stream one request and time it. TTFT is when the first chunk lands, and TPS comes from everything after. You can run that test in a few minutes against Together AI serverless inference, then repeat it at production concurrency.

FAQ

Does LLM inference need a GPU? Not for small models. Quantized models of a few billion parameters run on laptop CPUs with llama.cpp and GGUF files. For 70B-class models or many concurrent users you need one in practice, because decode speed tracks memory bandwidth and GPU memory has far more of it than system RAM.

What is the difference between AI inference and training? Training runs forward and backward passes and updates the weights. Inference runs forward passes only, with weights frozen, so there are no gradients or optimizer state to hold in memory. How large language models work covers training in full.

How expensive is LLM inference? It depends on model size, your mix of input and output tokens, and how you deploy. APIs price per million tokens, with output typically 3–4× input, and dedicated GPUs price per hour. The cost section works through an example.

Does quantization reduce output quality? Sometimes. fp8 is usually indistinguishable from 16-bit weights. 4-bit results vary by model, quantization method and task, so run your evals before switching a production endpoint.