Prompt engineering vs fine-tuning: when to tune
Prompt first and keep an eval set. Fine-tune only when prompts plateau or token cost at scale bites, and check that your provider still offers it.
Start with prompts. Fine-tune only when you have an eval set that shows the prompt has stopped improving, or when the cost of long prompts at scale is the problem. Google Cloud’s guidance says the same: “Start with prompting to find the optimal prompt. Then, move on to fine-tuning (if required)”. One more check applies in late 2026: some providers are closing fine-tuning, so confirm it is still available to you before planning around it.
This page is about prompting versus tuning. For the separate question of adding knowledge with retrieval, see RAG vs fine-tuning.
What changes
Prompt engineering changes the input: instructions, examples, structure. See prompt engineering and few-shot prompting. Fine-tuning changes the model. Google Cloud describes it as retraining the base model on your labeled data, adapting its weights.
Anthropic’s overview assumes you already have success criteria, a way to test them and a first draft prompt. OpenAI frames the work as a loop of evals, prompt engineering and fine-tuning. Without the evals you cannot tell which lever helped. See LLM evaluation.
Check availability first
OpenAI’s model optimization guide now says it is winding down its fine-tuning platform. The deprecations page gives the schedule, after a notice to developers on May 7, 2026:
| Date | What changes on OpenAI’s self-serve platform |
|---|---|
| May 7, 2026 | Organizations that never fine-tuned cannot create jobs |
| July 2, 2026 | Organizations with no inference on a fine-tuned model in the past 60 days cannot create jobs |
| January 6, 2027 | Active existing customers cannot create new jobs |
Inference on existing fine-tuned models continues until their base model is deprecated. Google Cloud’s Gemini Enterprise Agent Platform (formerly Vertex AI) still documents supervised, preference and reinforcement tuning for Gemini models. Whether a given model can be tuned changes, so read the provider’s current docs before you collect training data.
Where prompting falls short
OpenAI and Google Cloud name cases where prompts are not enough.
- More examples than fit in the context window. OpenAI lists this as a benefit of fine-tuning: you can provide more example inputs and outputs than fit in a single request.
- Token cost and latency at scale. OpenAI says shorter prompts with fewer examples can save token costs at scale and can be lower latency.
- A smaller, cheaper model. OpenAI says you can train a smaller model to excel at a task where a larger model is not cost-effective.
- Recurring errors after prompting. Google Cloud lists complex or unique tasks where advanced prompting strategies are not sufficient.
Before tuning, consider a cheaper lever. Anthropic notes that “you can sometimes improve latency and cost more easily by choosing a different model.”
Many-shot prompting as a middle path
Google DeepMind researchers Agarwal and colleagues submitted a paper in April 2024 on many-shot in-context learning, with hundreds or thousands of examples in the prompt. They report gains over few-shot prompting and that it “performs comparably to fine-tuning”, while “inference cost increases linearly” with the number of examples, and that frontier models benefit to varying degrees.
That cost point echoes older work. Liu and colleagues found in 2022 that parameter-efficient fine-tuning gave better accuracy and dramatically lower computational cost than few-shot in-context learning, because every prediction reprocesses the training examples. Those results used different models and tasks, so treat them as a reason to measure your own cost per request, not as a ranking.
Which kind of tuning
If you do tune, the method follows the failure. OpenAI and Google Cloud both offer these three families.
| Method | Trains on | Fits | Source |
|---|---|---|---|
| Supervised fine-tuning | Correct input and output examples | Strict output format, classification, extraction, instruction-following failures, style or tone | OpenAI, Google Cloud |
| Preference tuning (DPO) | Chosen and rejected responses to a prompt | Summaries that focus on the right content, conversational tone | OpenAI, Google Cloud |
| Reinforcement fine-tuning | A grader’s score on the result | Multi-step reasoning, verifiable tasks such as code or SQL | OpenAI, Google Cloud |
Google Cloud’s tuning guide gives typical sizes: hundreds to thousands of examples for supervised tuning, hundreds to thousands of preference pairs, and dozens to a few hundred prompts for reinforcement tuning. It adds that high-quality, well-labeled data beats quantity, and that you should evaluate where the model makes mistakes before adding more data. OpenAI limits reinforcement fine-tuning to reasoning models.
A decision order
- Write evals. Define success and score a baseline. See testing AI agents if the system calls tools.
- Improve the prompt. Clearer instructions, examples, structure. Prompt optimization covers doing this with data instead of by hand. If the issue is what the model sees across steps, see context engineering vs prompt engineering.
- Try a different model if the failing criterion is cost or latency.
- Read the failures that remain. Missing facts point to retrieval, not tuning. Format, tone or reasoning-style errors that survive prompting are what tuning targets.
- Confirm the provider still offers tuning for the model you need, and that you have enough labeled examples.
- Tune, then rerun the same evals. Keep the prompt-only version as the comparison. Tuning counts as a win only if the evals move more than a better prompt or model would have.
Run the same evals on both
Whichever way you go, the test set stays fixed. Run the prompt-only and tuned versions over the same cases and compare scores, cost and latency per request. If you trace production calls, the traces give you real failures to build that set from; HoneyHive’s docs show how to compare experiment runs side by side.
FAQ
Is prompt engineering or fine-tuning better?
Neither wins in general. Google Cloud's guidance is to start with prompting and move to fine-tuning only if needed, to boost performance or fix recurring errors.
How many examples do you need to fine-tune?
Google Cloud says to think 100 examples or more for Gemini, and that quality matters more than quantity. Its comparison table lists hundreds to thousands of labeled examples for supervised fine-tuning.
Can many examples in the prompt replace fine-tuning?
Sometimes. Google DeepMind researchers reported that many-shot prompting, with hundreds or thousands of examples, can perform comparably to fine-tuning, but inference cost grows linearly with the number of examples.
Can you still fine-tune OpenAI models?
Only if your organization already fine-tuned there and has run inference on a fine-tuned model in the past 60 days. OpenAI's deprecations page says new organizations were blocked on May 7, 2026, and that active existing customers lose the ability to create new jobs on January 6, 2027. Existing fine-tuned models keep serving until their base model is deprecated.
Does fine-tuning replace RAG?
No. They fix different problems: see RAG vs fine-tuning.