What is an LLM gateway?
An LLM gateway is a middleware layer between your app and model providers. What it does per request, what it cannot fix, and when you need one.
An LLM gateway is a service that sits between your application and one or more model providers. Your code sends every model request to the gateway, and the gateway authenticates it, applies limits, picks a provider, retries or falls back on failure, and logs the usage (OpenRouter, TrueFoundry). To the application it looks like a single API.
“AI gateway” and “LLM proxy” mean nearly the same thing in practice. The ranking explainers are written by gateway vendors, so read their feature lists as a map of what is possible, not as a verdict on which product to pick.
What a gateway does to a request
TrueFoundry describes six steps:
app -> gateway -> provider
1 intercept request
2 check key, role, rate limit
3 route to a provider or model
4 translate to provider format
5 normalize the response
6 log usage, latency, costStep 2 matters for cost: a request that fails the credential, permission or rate-limit check is rejected before any tokens are used. Step 6 is why gateways double as a record of who spent what, since usage can be attributed to a user, team or project.
What gateways offer
Feature sets differ, so check each one. These are the capabilities documented by the open-source LiteLLM gateway and Cloudflare’s AI Gateway.
| Capability | What it means | Documented in |
|---|---|---|
| One API | One OpenAI-compatible endpoint in front of many providers | LiteLLM |
| Routing | Pick a deployment by weight, latency, load or cost | LiteLLM routing |
| Retries and fallbacks | Try another model or provider on error; Cloudflare triggers a fallback “if a model request returns an error” by default | Cloudflare fallbacks |
| Cooldowns | LiteLLM sets a deployment aside after failures; the default cooldown is 5 seconds and triggers include 429 responses | LiteLLM routing |
| Keys and budgets | Virtual keys per user, team or project with their own models, budgets and rate limits | LiteLLM |
| Caching | Serve a repeated request from the gateway’s cache instead of the provider | Cloudflare |
| Rate limiting | Limit the number of requests your application receives | Cloudflare |
| Logs and analytics | Request counts, tokens and cost, plus logs of requests and errors | Cloudflare |
Guardrails such as masking PII before a request leaves your network are described in some vendor guides (OpenRouter), but not every gateway has them. Cloudflare’s overview page does not list them. See AI guardrails for what they check.
What a gateway cannot fix
- Failover is cleanest before a stream starts. If a provider fails mid-stream, the client may already hold partial output that your application has to handle (OpenRouter).
- A fallback model is a different model. The gateway can switch providers, but it cannot make two models answer the same way. Behavior, tool calling and output format can change, so a fallback needs the same testing as a primary.
- Retries need care. OpenRouter’s checklist lists retry idempotency as a criterion, so check what happens when a request is sent twice (OpenRouter).
- Quality is out of scope. A gateway reports that a call succeeded and what it cost. It does not tell you whether the answer was right.
Do you need one
This table is a rule of thumb built from the sources above, not part of any standard.
| Situation | Suggestion |
|---|---|
| One provider, one app | Call the provider directly (OpenRouter) |
| Several providers or models, and you want to switch without rewrites | A gateway gives one interface |
| Many teams sharing provider keys | A gateway with per-key budgets and limits |
| Provider rate limits or outages hurt you | A gateway with retries and fallbacks, tested against your prompts |
| Data must stay in your network | A self-hosted gateway; hosted ones route your requests through a third party |
OpenRouter’s guide gives five questions to ask first: must you self-host, do you need compliance certifications or data residency controls, how many providers and models you need, how much engineering time you can spend running infrastructure, and whether you need policy enforcement or dynamic routing. Gateway latency is not independently benchmarked in the sources read here, so test it yourself.
Gateways and tracing
A gateway sees each model call. It does not see the agent run around it: which step called the model, which tool ran, or how the final answer scored. Teams therefore pair a gateway with tracing. The OpenTelemetry standard is one way to carry the same trace data across tools. For the wider practice see LLM observability and agent observability.
Tracing gateway traffic in HoneyHive
HoneyHive’s docs have an AI Gateways section with integration pages for several AI gateways (docs index). For LiteLLM, the OpenInference instrumentor records completions with inputs, outputs and token usage, embedding and image generation calls, and model and provider metadata for each call. The docs advise using only the LiteLLM instrumentor for routed calls to avoid duplicate spans (LiteLLM integration).
FAQ
What is an LLM gateway?
A middleware layer that sits between your application and multiple model providers and handles authentication, rate limits, routing, failover and cost tracking behind one API (OpenRouter, TrueFoundry).
Is an LLM gateway the same as an LLM proxy?
Usage overlaps and vendors disagree. TrueFoundry calls a gateway "closely related to - but broader than" a proxy, adding routing intelligence, policy enforcement and observability. Treat the words as near-synonyms and compare features, not names.
Does an LLM gateway add latency?
Yes, one extra network hop. One vendor estimates about 3 to 10 ms of overhead (TrueFoundry, a vendor figure with no independent test). Model calls usually take far longer, so measure it in your own setup.
Do I need an LLM gateway if I use one provider?
Often not. OpenRouter's own guide recommends a direct API for single-provider use. A gateway starts to pay off with several providers, several teams sharing keys, or a need for central spend limits.
Does a gateway replace LLM observability?
No. Gateways log requests, tokens, latency and cost per call. Tracing a multi-step agent run and scoring output quality is a separate job, covered in LLM observability.