# What is vLLM?

> vLLM is an open-source engine for serving LLMs on your own GPUs. PagedAttention and continuous batching raise throughput, and it exposes an OpenAI-compatible API.

Source: https://www.honeyhive.ai/learn/vllm (updated 2026-11-19)

vLLM is an open-source library for running large language models on your own hardware. Its [documentation](https://docs.vllm.ai/en/latest/) calls it a fast and easy-to-use library for LLM inference and serving, and its [GitHub project](https://github.com/vllm-project/vllm) describes it as a high-throughput and memory-efficient inference and serving engine. It was originally developed in the Sky Computing Lab at UC Berkeley and uses the [Apache-2.0 license](https://github.com/vllm-project/vllm).

If you call a hosted model API, the provider runs a stack like this for you. vLLM is what you run when you host the model yourself. For the basics of what that stack does, see [LLM inference](/learn/llm-inference).

## What problem it solves

In serving, GPU memory often limits how many requests run at once. Each request keeps a [KV cache](/learn/kv-cache) that grows and shrinks as tokens are generated. The vLLM authors [wrote](https://arxiv.org/abs/2309.06180) that fragmentation and redundant duplication of this cache waste memory and limit batch size. Their answer, PagedAttention, is an attention algorithm inspired by virtual memory and paging in operating systems.

The [paper](https://arxiv.org/abs/2309.06180) reports that vLLM improves the throughput of popular LLMs by 2 to 4 times at the same latency as FasterTransformer and Orca, with larger gains for longer sequences, larger models and more complex decoding. It also reports near-zero waste in KV cache memory and cache sharing within and across requests. These are 2023 results against the systems of that time, so benchmark on your own model and traffic.

## What it includes today

The PyTorch Foundation [welcomed](https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/) vLLM as a hosted project in May 2025, and noted it had grown from PagedAttention into a full inference engine. The [docs](https://docs.vllm.ai/en/latest/) list these features:

| Area | What the docs list |
|---|---|
| Scheduling and memory | PagedAttention, continuous batching, chunked prefill, prefix caching |
| Quantization | FP8, INT8, INT4, GPTQ/AWQ, GGUF and more |
| Faster decoding | Speculative decoding, including n-gram and EAGLE |
| Distributed inference | Tensor, pipeline, data, expert and context parallelism |
| Output control | Structured outputs with xgrammar or guidance |
| Adapters | Multi-LoRA support |
| Hardware | NVIDIA and AMD GPUs, x86, ARM and PowerPC CPUs, plus plugins for TPUs, Gaudi and others |

The [GitHub project](https://github.com/vllm-project/vllm) says vLLM supports 200+ model architectures on Hugging Face, which matters if you run [open-weight models](/learn/open-weight-models). [Prefix caching](/learn/prefix-caching) is the self-hosted counterpart of [prompt caching](/learn/prompt-caching) on hosted APIs, and structured outputs are covered in [structured output](/learn/structured-output-llm).

## Two ways to run it

The [quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart/) requires Linux and Python 3.11 through 3.14, and recommends `uv` for environments.

**Offline batch inference.** Create an engine in your own Python process and pass a list of prompts:

```python
from vllm import LLM, SamplingParams

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)
```

The docs warn that `llm.generate` does not apply a model's chat template, so for chat models apply it yourself or use `llm.chat` with a list of messages.

**Online serving.** Start an HTTP server:

```bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```

The quickstart [says](https://docs.vllm.ai/en/latest/getting_started/quickstart/) it listens on `http://localhost:8000` by default, and `--host` and `--port` change that. The [serving docs](https://docs.vllm.ai/en/latest/serving/online_serving/) list OpenAI-compatible routes including `/v1/completions`, `/v1/chat/completions`, `/v1/responses`, `/v1/embeddings` and audio transcription and translation. The [docs](https://docs.vllm.ai/en/latest/) also list the Anthropic Messages API and gRPC support. Because the interface matches, an app that uses the OpenAI client can switch by setting the base URL to `http://localhost:8000/v1` and a placeholder key such as `EMPTY`.

## When it fits

Red Hat's [overview](https://www.redhat.com/en/topics/ai/what-is-vllm) says vLLM suits teams that want maximum performance from their servers, are moving to large-scale production, or want to reduce GPU count or self-host for more control over data privacy. The cost is operations: you manage GPUs, upgrades, scaling and capacity, which a hosted API does for you. Other open-source engines exist, such as SGLang and TensorRT-LLM, and [LLM inference](/learn/llm-inference) compares how each project describes itself. For spend on hosted APIs, see [LLM cost optimization](/learn/llm-cost-optimization).

## What to watch once it is running

The server exposes a [Prometheus-compatible](https://docs.vllm.ai/en/latest/serving/online_serving/) `/metrics` endpoint for engine health. What your users feel is request-level: [time to first token](/learn/time-to-first-token), total latency, errors and token counts per call. Those belong in your application traces, next to the prompt and the response, as described in [LLM observability](/learn/llm-observability). HoneyHive's [tracing quickstart](https://docs.honeyhive.ai/v2/introduction/tracing-quickstart.md) shows instrumenting an OpenAI call and inspecting the full request and response, which is the same client shape vLLM's compatible endpoint accepts.

## FAQ

### What is vLLM?

vLLM is, in its [documentation](https://docs.vllm.ai/en/latest/), a fast and easy-to-use library for LLM inference and serving. It was originally developed in the Sky Computing Lab at UC Berkeley and is released under the [Apache-2.0 license](https://github.com/vllm-project/vllm).

### What is PagedAttention?

It is the attention algorithm behind vLLM. The [paper](https://arxiv.org/abs/2309.06180) says it is inspired by virtual memory and paging in operating systems and manages the KV cache in blocks, which cuts wasted memory.

### Is vLLM compatible with the OpenAI API?

Yes. The vLLM [quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart/) says to point the OpenAI Python client at `http://localhost:8000/v1` with a placeholder API key such as `EMPTY` when no key is configured.

### Is vLLM only for NVIDIA GPUs?

No. Its [docs](https://docs.vllm.ai/en/latest/) list NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs, with hardware plugins for other accelerators such as Google TPUs and Intel Gaudi.

### Can vLLM run without a server?

Yes. The same [quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart/) shows offline batched inference with the `LLM` class and `llm.generate`, besides `vllm serve` for an HTTP server.
