# What is LLMOps?

> LLMOps is the set of practices for running LLM applications: versioning prompts, evals, tracing, monitoring, cost control and handling model changes.

Source: https://www.honeyhive.ai/learn/llmops (updated 2026-11-07)

LLMOps (large language model operations) is the set of practices for managing LLMs and the applications built on them after the first demo works. Google Cloud [defines](https://cloud.google.com/discover/what-is-llmops) it as the practices and processes involved in managing and operating large language models, and Red Hat [puts it](https://redhat.com/en/topics/ai/llmops) as "operational methods used to manage large language models."

In practice it covers the work that keeps an LLM app good after release: versioning prompts, running [evaluations](/learn/llm-evaluation), recording [traces](/learn/llm-observability), [monitoring](/learn/llm-monitoring) production, controlling cost and handling model changes.

## How it relates to MLOps

The [MLOps vs LLMOps](/learn/mlops-vs-llmops) page compares the two side by side.

Google Cloud [calls](https://cloud.google.com/discover/what-is-llmops) LLMOps a specialized subset of MLOps. Its MLOps guide [defines](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) MLOps as an engineering culture and practice that unifies ML system development and operation, and [names](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) continuous training, the automatic retraining and serving of models, as a property unique to ML systems.

Red Hat [lists](https://redhat.com/en/topics/ai/llmops) the differences that matter for LLMs:

| Area | Traditional ML | LLMs, per Red Hat |
|---|---|---|
| Starting point | Usually created or trained from scratch | Start from a foundation model and are fine-tuned |
| Feedback | Specific accuracy metrics | Feedback is used to evaluate accuracy, and feedback from humans (RLHF) is critical to performance |
| Quality measures | Precise metrics | Text-quality measures such as [BLEU](/learn/bleu-score) and [ROUGE](/learn/rouge-score), which need more complex evaluation |

## Two readings of the term

The vendor definitions describe a training-centered lifecycle. Red Hat's [six stages](https://redhat.com/en/topics/ai/llmops) are exploratory data analysis, data prep and prompt engineering, fine-tuning, review and governance, inference and serving, and monitoring with human feedback. Google Cloud's [activity list](https://cloud.google.com/discover/what-is-llmops) includes data management, training and fine-tuning, deployment, monitoring and evaluation, and security and compliance.

Many teams build on a hosted model and never train one. Their loop is the one OpenAI's [optimization guide](https://developers.openai.com/api/docs/guides/model-optimization) describes: tweak the prompt or fine-tuning dataset based on eval feedback, a cycle it calls a flywheel that leads to better prompts and better training data. For them the operational work is mostly about prompts, evals, traces, cost and model changes, and the next sections cover those.

## What the work consists of

| Task | What the sources say |
|---|---|
| Version prompts | Chip Huyen [wrote](https://huyenchip.com/2023/04/11/llm-engineering.html) in April 2023 that it is essential to version and track the performance of each prompt, and suggested git. She [noted](https://huyenchip.com/2023/04/11/llm-engineering.html) that an accidentally changed prompt still runs but gives very different outputs. |
| Test before shipping | OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices) eval-driven development: evaluate early and often, with scoped tests at every stage. Hamel Husain [describes](https://hamel.dev/blog/posts/evals/) three levels: unit-test style assertions, human and model evaluation, and A/B testing. |
| Record what happens | OpenAI [says](https://developers.openai.com/api/docs/guides/evaluation-best-practices) to log everything so you can mine the logs for eval cases. Husain [writes](https://hamel.dev/blog/posts/evals/) that searching, filtering and reading traces are essential features of a logging tool. |
| Run evals on every change | OpenAI [recommends](https://developers.openai.com/api/docs/guides/evaluation-best-practices) continuous evaluation, running evals on every change. |
| Watch production | Anthropic's engineers [found](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) in January 2026 that the most effective teams combine automated evals for fast iteration with production monitoring for ground truth. Google Cloud [suggests](https://cloud.google.com/discover/what-is-llmops) KPIs such as accuracy, latency and resource utilization. |
| Control cost and latency | Huyen [wrote](https://huyenchip.com/2023/04/11/llm-engineering.html) that "the cost of LLMOps is in inference" and that output length significantly affects latency. See [LLM cost optimization](/learn/llm-cost-optimization). |
| Handle model changes | OpenAI [notes](https://developers.openai.com/api/docs/guides/model-optimization) that model behavior changes between snapshots and families. Anthropic [retires](https://platform.claude.com/docs/en/about-claude/model-deprecations) older models, gives at least 60 days' notice for publicly released ones, and says requests to retired models fail. |
| Govern and secure | Red Hat [includes](https://redhat.com/en/topics/ai/llmops) a review and governance stage, and Google Cloud [lists](https://cloud.google.com/discover/what-is-llmops) security and compliance. See [AI guardrails](/learn/ai-guardrails) and [LLM security](/learn/llm-security). |

## Why prompts and models need this treatment

OpenAI's guide [states](https://developers.openai.com/api/docs/guides/evaluation-best-practices) that generative AI is variable and that models sometimes produce different output from the same input. Without evals, Anthropic's engineers [say](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), teams get stuck in reactive loops and catch issues only in production, and they cannot tell real regressions from noise.

Huyen [argued](https://huyenchip.com/2023/04/11/llm-engineering.html) that if you expect the models you use to change, you should unit-test all your prompts using evaluation examples. Her post dates from April 2023. OpenAI and Anthropic's guidance since then reaches the same practice.

## Where to start

An order that builds on itself, drawn from the guidance above:

1. Log every request, as OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices).
2. Read the logs. Husain [says](https://hamel.dev/blog/posts/evals/) you must remove all friction from the process of looking at data.
3. Turn what you find into eval cases, and run them on every change. See [error analysis](/learn/error-analysis-llm) and [LLM evaluation](/learn/llm-evaluation).
4. Monitor production, and check that your graders work by reading transcripts and grades from many trials, as Anthropic [recommends](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents).
5. Check your model's retirement date before it becomes an outage, using the [export](https://platform.claude.com/docs/en/about-claude/model-deprecations) of usage by model that Anthropic provides.

For improving the prompts themselves once you have evals, see [prompt optimization](/learn/prompt-optimization). For choosing between changing the prompt, adding retrieval or fine-tuning, see [RAG vs fine-tuning](/learn/rag-vs-fine-tuning).

## Set it up in HoneyHive

HoneyHive's [tracing concepts](https://docs.honeyhive.ai/v2/tracing/concepts.md) describe a session as the root of each event tree, with model, tool and chain events under it. Its [CI regression detection guide](https://docs.honeyhive.ai/v2/evaluation/ci-regression-detection.md) fails the build when a code change degrades a metric beyond your threshold. Its [monitoring docs](https://docs.honeyhive.ai/v2/monitoring/overview.md) cover charts, online evaluations on production traces and alerts.

## FAQ

### What is LLMOps?

LLMOps is the practice of managing large language models and the applications built on them across their lifecycle. Google Cloud [defines](https://cloud.google.com/discover/what-is-llmops) it as the practices and processes involved in managing and operating LLMs.

### What is the difference between LLMOps and MLOps?

Google Cloud [calls](https://cloud.google.com/discover/what-is-llmops) LLMOps a specialized subset of MLOps. Red Hat [adds](https://redhat.com/en/topics/ai/llmops) that LLMs start from a foundation model and are fine-tuned, while traditional ML models are usually trained from scratch.

### What are the stages of LLMOps?

Red Hat [lists](https://redhat.com/en/topics/ai/llmops) six: exploratory data analysis, data prep and prompt engineering, fine-tuning, review and governance, inference and serving, and monitoring with human feedback. Teams that call a hosted model skip the training stages but keep the rest.

### Why do LLM apps need their own operations practice?

Outputs vary and prompts fail silently. OpenAI's guide [says](https://developers.openai.com/api/docs/guides/evaluation-best-practices) models sometimes produce different output from the same input, and Chip Huyen [wrote](https://huyenchip.com/2023/04/11/llm-engineering.html) that a prompt changed by accident still runs but gives very different outputs.

### How do you start with LLMOps?

Log every request, turn the logs into eval cases, run the evals on every change and monitor production. OpenAI [recommends](https://developers.openai.com/api/docs/guides/evaluation-best-practices) logging everything and running continuous evaluation on every change.