What is LLM inference?
LLM inference is running a trained model to generate output tokens: a parallel prefill step, then a token-by-token decode step that sets cost and latency.
LLM inference is running a pre-trained large language model to generate output tokens for a new prompt. The model’s parameters do not change. Every chat reply, tool call and agent step you send to a model is one inference request.
Inference versus training
IBM contrasts the two. Training includes expensive steps such as backpropagation and keeps updating weights. Inference is a read-only, forward-only pass through weights that are already fixed. Because inference runs continuously once an app is live, IBM says its efficiency directly affects infrastructure cost.
Prefill and decode
NVIDIA’s inference guide splits a request into two phases that behave differently.
| Phase | What happens | Hardware behavior (NVIDIA) |
|---|---|---|
| Prefill | The model processes the whole input prompt at once | A highly parallelized matrix-matrix operation that effectively saturates GPU utilization |
| Decode | The model generates output one token at a time until a stop condition | A matrix-vector operation that underutilizes GPU compute and is memory-bound |
The split explains a lot of latency behavior. A long prompt lengthens prefill, and OpenAI notes that a long prompt can delay the first token, especially when much of the input is new. A long answer lengthens decode. OpenAI also advises looking at what the model generates when an answer starts quickly but finishes slowly. The metrics for each phase are covered in time to first token.
Decode also reads the attention keys and values of every earlier token. Servers keep them in GPU memory in a KV cache, and NVIDIA says that cache grows linearly with batch size and sequence length, so memory can run out quickly.
Batching
Because decode underuses the GPU, serving many requests together raises throughput. The vLLM authors wrote that high-throughput serving requires batching sufficiently many requests at a time.
The Orca paper by Yu and colleagues proposed iteration-level scheduling, which schedules work at the granularity of one iteration. Serving engines now commonly offer the result as continuous or in-flight batching: NVIDIA describes the runtime evicting finished sequences from the batch immediately so new requests can start.
Hosted APIs and self-hosted engines
With a hosted API, the provider runs the serving stack and you send requests to it. Self-hosting means you run an engine on your own GPUs. Three open-source engines are:
| Engine | How its project describes it |
|---|---|
| vLLM | A high-throughput and memory-efficient inference and serving engine, with PagedAttention, continuous batching, chunked prefill, prefix caching and an OpenAI-compatible API server. Started in UC Berkeley’s Sky Computing Lab |
| SGLang | An open-source inference framework for language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts |
| TensorRT-LLM | An open-source library for optimizing inference on NVIDIA GPUs, with in-flight batching, a paged KV cache and FP4 and FP8 quantization |
PagedAttention, the memory technique behind vLLM, reported near-zero KV cache waste and 2 to 4 times the throughput of FasterTransformer and Orca at similar latency, in the authors’ 2023 tests. Engines have changed since, so benchmark any candidate on your own model and traffic.
What it means for cost and latency
IBM lists a four-way tradeoff in inference optimization: accuracy, speed, memory use and infrastructure cost. A few practical consequences follow from the phases above:
- Prompt length drives prefill time and the first-token wait. See context window.
- Output length drives decode time, so a long answer finishes slowly even when it starts fast. OpenAI’s latency guide suggests generating fewer tokens in that case.
- Reusing a shared prompt prefix saves work. Anthropic says prompt caching significantly reduces processing time and costs for prompts with consistent elements. Related ideas are semantic caching, model routing and the wider list in LLM cost optimization.
- Under load, queuing adds to the first-token wait. NVIDIA says time to first token generally includes queuing, prefill and network latency.
Measure it in your own app
You see inference through its effects: token counts, call duration and cost per request. HoneyHive’s monitoring dashboard tracks cost, latency, token usage and evaluator scores, so a change in prompt length or model shows up next to quality. For the wider practice, see LLM monitoring.
FAQ
What is LLM inference?
It is the process of running a pre-trained large language model to generate output tokens for new input prompts, with no update to the model's parameters, as IBM defines it.
What is the difference between prefill and decode?
Prefill processes the whole prompt in parallel and keeps the GPU busy. Decode then produces one token at a time and is memory-bound.
What is continuous batching?
It is a scheduling technique where the server removes finished sequences from the batch right away and starts new requests, instead of waiting for the whole batch to end, as NVIDIA describes.
Is inference the same as training?
No. IBM describes training as computationally expensive, with steps such as backpropagation, and inference as a read-only, forward-only process.
Which engines serve LLMs on your own hardware?
vLLM, SGLang and TensorRT-LLM are three open-source options. vLLM calls itself an inference and serving engine, SGLang an inference framework and TensorRT-LLM an open-source library for inference on NVIDIA GPUs.