What is vLLM?
vLLM is an open-source engine for serving LLMs on your own GPUs. PagedAttention and continuous batching raise throughput, and it exposes an OpenAI-compatible API.
vLLM is an open-source library for running large language models on your own hardware. Its documentation calls it a fast and easy-to-use library for LLM inference and serving, and its GitHub project describes it as a high-throughput and memory-efficient inference and serving engine. It was originally developed in the Sky Computing Lab at UC Berkeley and uses the Apache-2.0 license.
If you call a hosted model API, the provider runs a stack like this for you. vLLM is what you run when you host the model yourself. For the basics of what that stack does, see LLM inference.
What problem it solves
In serving, GPU memory often limits how many requests run at once. Each request keeps a KV cache that grows and shrinks as tokens are generated. The vLLM authors wrote that fragmentation and redundant duplication of this cache waste memory and limit batch size. Their answer, PagedAttention, is an attention algorithm inspired by virtual memory and paging in operating systems.
The paper reports that vLLM improves the throughput of popular LLMs by 2 to 4 times at the same latency as FasterTransformer and Orca, with larger gains for longer sequences, larger models and more complex decoding. It also reports near-zero waste in KV cache memory and cache sharing within and across requests. These are 2023 results against the systems of that time, so benchmark on your own model and traffic.
What it includes today
The PyTorch Foundation welcomed vLLM as a hosted project in May 2025, and noted it had grown from PagedAttention into a full inference engine. The docs list these features:
| Area | What the docs list |
|---|---|
| Scheduling and memory | PagedAttention, continuous batching, chunked prefill, prefix caching |
| Quantization | FP8, INT8, INT4, GPTQ/AWQ, GGUF and more |
| Faster decoding | Speculative decoding, including n-gram and EAGLE |
| Distributed inference | Tensor, pipeline, data, expert and context parallelism |
| Output control | Structured outputs with xgrammar or guidance |
| Adapters | Multi-LoRA support |
| Hardware | NVIDIA and AMD GPUs, x86, ARM and PowerPC CPUs, plus plugins for TPUs, Gaudi and others |
The GitHub project says vLLM supports 200+ model architectures on Hugging Face, which matters if you run open-weight models. Prefix caching is the self-hosted counterpart of prompt caching on hosted APIs, and structured outputs are covered in structured output.
Two ways to run it
The quickstart requires Linux and Python 3.11 through 3.14, and recommends uv for environments.
Offline batch inference. Create an engine in your own Python process and pass a list of prompts:
from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)
The docs warn that llm.generate does not apply a model’s chat template, so for chat models apply it yourself or use llm.chat with a list of messages.
Online serving. Start an HTTP server:
vllm serve Qwen/Qwen2.5-1.5B-Instruct
The quickstart says it listens on http://localhost:8000 by default, and --host and --port change that. The serving docs list OpenAI-compatible routes including /v1/completions, /v1/chat/completions, /v1/responses, /v1/embeddings and audio transcription and translation. The docs also list the Anthropic Messages API and gRPC support. Because the interface matches, an app that uses the OpenAI client can switch by setting the base URL to http://localhost:8000/v1 and a placeholder key such as EMPTY.
When it fits
Red Hat’s overview says vLLM suits teams that want maximum performance from their servers, are moving to large-scale production, or want to reduce GPU count or self-host for more control over data privacy. The cost is operations: you manage GPUs, upgrades, scaling and capacity, which a hosted API does for you. Other open-source engines exist, such as SGLang and TensorRT-LLM, and LLM inference compares how each project describes itself. For spend on hosted APIs, see LLM cost optimization.
What to watch once it is running
The server exposes a Prometheus-compatible /metrics endpoint for engine health. What your users feel is request-level: time to first token, total latency, errors and token counts per call. Those belong in your application traces, next to the prompt and the response, as described in LLM observability. HoneyHive’s tracing quickstart shows instrumenting an OpenAI call and inspecting the full request and response, which is the same client shape vLLM’s compatible endpoint accepts.
FAQ
What is vLLM?
vLLM is, in its documentation, a fast and easy-to-use library for LLM inference and serving. It was originally developed in the Sky Computing Lab at UC Berkeley and is released under the Apache-2.0 license.
What is PagedAttention?
It is the attention algorithm behind vLLM. The paper says it is inspired by virtual memory and paging in operating systems and manages the KV cache in blocks, which cuts wasted memory.
Is vLLM compatible with the OpenAI API?
Yes. The vLLM quickstart says to point the OpenAI Python client at http://localhost:8000/v1 with a placeholder API key such as EMPTY when no key is configured.
Is vLLM only for NVIDIA GPUs?
No. Its docs list NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs, with hardware plugins for other accelerators such as Google TPUs and Intel Gaudi.
Can vLLM run without a server?
Yes. The same quickstart shows offline batched inference with the LLM class and llm.generate, besides vllm serve for an HTTP server.