# What is LLM inference?

> LLM inference is running a trained model to generate output tokens: a parallel prefill step, then a token-by-token decode step that sets cost and latency.

Source: https://www.honeyhive.ai/learn/llm-inference (updated 2026-11-16)

LLM inference is [running a pre-trained large language model](https://www.ibm.com/think/topics/llm-inference) to generate output tokens for a new prompt. The model's parameters do not change. Every chat reply, tool call and agent step you send to a model is one inference request.

## Inference versus training

IBM [contrasts](https://www.ibm.com/think/topics/llm-inference) the two. Training includes expensive steps such as backpropagation and keeps updating weights. Inference is a read-only, forward-only pass through weights that are already fixed. Because inference runs continuously once an app is live, IBM says its efficiency directly affects infrastructure cost.

## Prefill and decode

NVIDIA's [inference guide](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) splits a request into two phases that behave differently.

| Phase | What happens | Hardware behavior (NVIDIA) |
|---|---|---|
| Prefill | The model processes the whole input prompt at once | A highly parallelized matrix-matrix operation that effectively saturates GPU utilization |
| Decode | The model generates output one token at a time until a stop condition | A matrix-vector operation that underutilizes GPU compute and is memory-bound |

The split explains a lot of latency behavior. A long prompt lengthens prefill, and OpenAI [notes](https://developers.openai.com/api/docs/guides/latency-optimization) that a long prompt can delay the first token, especially when much of the input is new. A long answer lengthens decode. OpenAI also [advises](https://developers.openai.com/api/docs/guides/latency-optimization) looking at what the model generates when an answer starts quickly but finishes slowly. The metrics for each phase are covered in [time to first token](/learn/time-to-first-token).

Decode also reads the attention keys and values of every earlier token. Servers keep them in GPU memory in a [KV cache](/learn/kv-cache), and NVIDIA [says](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) that cache grows linearly with batch size and sequence length, so memory can run out quickly.

## Batching

Because decode underuses the GPU, serving many requests together raises throughput. The [vLLM](/learn/vllm) authors [wrote](https://arxiv.org/abs/2309.06180) that high-throughput serving requires batching sufficiently many requests at a time.

The Orca paper by Yu and colleagues [proposed](https://www.usenix.org/conference/osdi22/presentation/yu) iteration-level scheduling, which schedules work at the granularity of one iteration. Serving engines now commonly offer the result as continuous or in-flight batching: NVIDIA [describes](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) the runtime evicting finished sequences from the batch immediately so new requests can start.

## Hosted APIs and self-hosted engines

With a hosted API, the provider runs the serving stack and you send requests to it. Self-hosting means you run an engine on your own GPUs. Three open-source engines are:

| Engine | How its project describes it |
|---|---|
| [vLLM](https://github.com/vllm-project/vllm) | A high-throughput and memory-efficient inference and serving engine, with PagedAttention, continuous batching, chunked prefill, prefix caching and an OpenAI-compatible API server. Started in UC Berkeley's Sky Computing Lab |
| [SGLang](https://github.com/sgl-project/sglang) | An open-source inference framework for language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts |
| [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/overview.html) | An open-source library for optimizing inference on NVIDIA GPUs, with in-flight batching, a paged KV cache and FP4 and FP8 quantization |

PagedAttention, the memory technique behind vLLM, [reported](https://arxiv.org/abs/2309.06180) near-zero KV cache waste and 2 to 4 times the throughput of FasterTransformer and Orca at similar latency, in the authors' 2023 tests. Engines have changed since, so benchmark any candidate on your own model and traffic.

## What it means for cost and latency

IBM [lists](https://www.ibm.com/think/topics/llm-inference) a four-way tradeoff in inference optimization: accuracy, speed, memory use and infrastructure cost. A few practical consequences follow from the phases above:

- Prompt length drives prefill time and the first-token wait. See [context window](/learn/context-window).
- Output length drives decode time, so a long answer finishes slowly even when it starts fast. OpenAI's latency guide [suggests](https://developers.openai.com/api/docs/guides/latency-optimization) generating fewer tokens in that case.
- Reusing a shared prompt prefix saves work. Anthropic says prompt caching [significantly reduces](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) processing time and costs for prompts with consistent elements. Related ideas are [semantic caching](/learn/semantic-caching), [model routing](/learn/model-routing) and the wider list in [LLM cost optimization](/learn/llm-cost-optimization).
- Under load, queuing adds to the first-token wait. NVIDIA [says](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) time to first token generally includes queuing, prefill and network latency.

## Measure it in your own app

You see inference through its effects: token counts, call duration and cost per request. HoneyHive's monitoring dashboard [tracks](https://docs.honeyhive.ai/v2/monitoring/overview.md) cost, latency, token usage and evaluator scores, so a change in prompt length or model shows up next to quality. For the wider practice, see [LLM monitoring](/learn/llm-monitoring).

## FAQ

### What is LLM inference?

It is the process of running a pre-trained large language model to generate output tokens for new input prompts, with no update to the model's parameters, as IBM [defines it](https://www.ibm.com/think/topics/llm-inference).

### What is the difference between prefill and decode?

Prefill processes the whole prompt in parallel and keeps the GPU busy. Decode then produces one token at a time and is [memory-bound](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/).

### What is continuous batching?

It is a scheduling technique where the server removes finished sequences from the batch right away and starts new requests, instead of waiting for the whole batch to end, as NVIDIA [describes](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/).

### Is inference the same as training?

No. IBM describes training as computationally expensive, with steps such as backpropagation, and inference as a [read-only, forward-only process](https://www.ibm.com/think/topics/llm-inference).

### Which engines serve LLMs on your own hardware?

vLLM, SGLang and TensorRT-LLM are three open-source options. vLLM calls itself an [inference and serving engine](https://github.com/vllm-project/vllm), SGLang an [inference framework](https://github.com/sgl-project/sglang) and TensorRT-LLM an [open-source library](https://nvidia.github.io/TensorRT-LLM/overview.html) for inference on NVIDIA GPUs.
