MLOps vs LLMOps
MLOps runs the training and serving of your own models. LLMOps runs apps built on a foundation model: prompts, evals, traces and inference cost.
MLOps is the practice of building, deploying and running machine learning models you train yourself. LLMOps is the narrower practice of running applications built on large language models, where the model often comes from a provider and most of the work is prompts, evaluation and inference. Google Cloud calls LLMOps “a specialized subset of MLOps,” so the two overlap more than the names suggest.
For a definition of the newer term on its own, see what is LLMOps.
What MLOps covers
Google Cloud defines MLOps as “an ML engineering culture and practice that aims at unifying ML system development (Dev) and ML system operation (Ops).” The same guide lists what changes compared with ordinary software:
- Continuous training, which it calls “a new property, unique to ML systems,” automatically retrains and serves models.
- Testing needs data validation on top of unit and integration tests.
- Production needs tracking of summary statistics of the data and the online performance of the model.
- The model is retrained when monitoring shows noticeable performance degradation.
Everything in that list assumes a team owns a training pipeline and a labeled dataset.
Where the work differs
| Area | MLOps | LLMOps |
|---|---|---|
| Where the model comes from | Red Hat says traditional models are usually created or trained from scratch | LLMs begin with a foundation model and are fine-tuned on data (Red Hat) |
| Main way to change behavior | Retraining on new data, which Google Cloud calls continuous training | Prompt changes. Huyen wrote that prompt engineering is a cheap and fast way to get something running |
| What needs versioning | Data, data schemas and models, which Google Cloud says CI must also test and validate, with pipeline versions and artifact lineage recorded in a metadata store | Also prompts. Huyen: it is “essential to version and track the performance of each prompt.” See prompt versioning |
| How it fails | Performance degrades and the model goes stale, which monitoring detects (Google Cloud) | A changed prompt “will still run but give very different outputs” (Huyen), and Huyen adds there is no way to guarantee prompts still work as intended with a newer model |
| Evaluation | Specific accuracy metrics (Red Hat) | Feedback-based evaluation plus measures such as BLEU and ROUGE that need more complex evaluation (Red Hat). See LLM evaluation |
| Variability | Not addressed in Google Cloud’s MLOps guide | OpenAI states that models sometimes produce different output from the same input |
| Where the cost sits | Collecting data and training, usually much higher and slower than prompting (Huyen) | “The cost of LLMOps is in inference” (Huyen). See LLM cost optimization |
| Human feedback | Not addressed in Google Cloud’s MLOps guide | Red Hat calls feedback from humans, including RLHF, critical to LLM performance |
Huyen’s post dates from April 2023, so treat the cost comparison as her view at the time.
What carries over
Deployment, serving, versioning and monitoring exist in both. Google Cloud’s LLMOps page lists model deployment and maintenance, data management, training and fine-tuning, monitoring and evaluation, and security and compliance as LLMOps activities, which is close to the MLOps list. What changes is what you version and what you measure.
Monitoring is the clearest case. MLOps monitoring watches the data and the model for staleness. For an LLM app the same idea applies to prompts and outputs. OpenAI recommends monitoring the app to identify new cases of nondeterminism. See LLM monitoring and data drift vs concept drift for how drift shows up in LLM apps.
Which practice you need
| Your situation | What to run | Source |
|---|---|---|
| You train or retrain models on your own data | The MLOps pipeline: data validation, continuous training, model validation, monitoring for staleness | Google Cloud |
| You fine-tune a foundation model | MLOps pipeline pieces for data and training, plus prompt and eval practices | Red Hat |
| You call a hosted model with prompts | Prompt versions, logging, evals on every change, monitoring, cost tracking | OpenAI, Huyen |
| You do both (a classic model plus an LLM feature) | Both sets, since the failure modes differ | The two rows above |
Evaluation is the sharpest break
In MLOps, Google Cloud’s guide evaluates the model on a holdout test set and confirms it beats a baseline before deployment. For LLM apps, OpenAI’s guide says “Generative AI is variable,” so it asks teams to log as they develop, mine the logs for eval cases and run evals on every change. It adds that “Evaluation is a continuous process.”
Huyen wrote that there is no way to guarantee all your prompts still work as intended with a newer model, so it is important to unit-test all your prompts using evaluation examples. See error analysis for turning logs into eval cases.
Setting up the LLM side in HoneyHive
HoneyHive’s tracing concepts describe a session as the root of each trace tree, with model, tool and chain events nested under it. Its evaluation docs describe offline experiments that run a function over a dataset to compare prompts, models and pipelines before deploy. The monitoring overview covers charts, online evaluations on production traces and alerts.
FAQ
What is the difference between MLOps and LLMOps?
MLOps is an engineering practice that unifies ML system development and operation, as Google Cloud defines it. LLMOps applies that to large language models, which Google Cloud calls a specialized subset of MLOps. The day to day work shifts from data and training pipelines toward prompts, evaluation and inference.
Is LLMOps replacing MLOps?
Not according to Google Cloud, which describes LLMOps as a subset of MLOps, so the practices for deployment and monitoring carry over. Teams that train their own models still need the MLOps pipeline.
Do I need LLMOps if I only call a hosted model?
You need the parts that run after release. OpenAI says models sometimes produce different output from the same input and recommends running evals on every change. You can skip training pipelines, but not prompt versions, evals, logging or cost tracking.
Why is LLM evaluation different from classic ML evaluation?
Red Hat says traditional ML models use specific accuracy metrics, while LLMs use measures such as BLEU and ROUGE that need more complex evaluation. See LLM evaluation for the methods.
Where does the cost sit in LLMOps?
Chip Huyen wrote that the traditional cost of collecting data and training models is usually much higher, but that for LLM apps "the cost of LLMOps is in inference."