Prompt optimization
Prompt optimization improves a prompt against a metric, by hand or with tools like DSPy, GEPA and provider optimizers. What each method needs and where it fails.
Prompt optimization is improving a prompt by measuring it against a metric and keeping the versions that score higher. You can do it by hand, with a provider’s rewriting tool, or with an algorithm that searches over instructions and examples. All three depend on the same thing: a way to score outputs. Without that, you are only rewording. For the writing side, see what is prompt engineering and prompt engineering techniques.
Three ways to optimize a prompt
| Approach | What changes the prompt | What it needs |
|---|---|---|
| Manual iteration | You edit one thing at a time and re-run your tests | Test inputs and a way to grade outputs (LLM evaluation) |
| Provider rewriters | A tool rewrites your prompt to the provider’s guidance (OpenAI’s dataset-backed version is being deprecated; Anthropic’s was retired) | A prompt; optionally feedback and examples |
| Algorithmic optimizers | A search proposes and scores candidate instructions and examples | A program, a metric and training inputs |
Provider rewriting tools
These tools restructure a prompt using the vendor’s own guidance. Anthropic’s is gone and OpenAI’s dataset-backed version is being shut down, so treat them as a pattern more than a dependency.
- OpenAI. The prompt optimizer is a dashboard chat interface where you enter a prompt and it is optimized “according to current best practices”. Paired with a dataset, it can use Good/Bad annotations, text critiques and grader results. OpenAI says a dataset needs at least three rows with model responses and at least one grader result or annotation per row. It adds that effectiveness depends on narrowly defined graders. The dataset-backed version is being deprecated with the Evals platform: Evals became read-only for existing users in October 2026 and the platform is scheduled to shut down in November 2026.
- Anthropic (retired). Anthropic offered a Console prompt improver. Its release notes say the legacy Workbench was sunset with access ending in August 2026, and the experimental prompt tools APIs, including
/v1/experimental/improve_prompt, were retired the same day. When it was available, Anthropic reported a 30% accuracy increase on a multilabel classification test in its own testing.
Algorithmic optimizers
DSPy is a widely used open-source toolkit for this. Its optimizers tune the prompts and/or the LM weights of a program. You give them a program, a metric that scores output (higher is better), and training inputs, which can be as few as 5 or 10 examples without labels.
| Optimizer | What it does | Source |
|---|---|---|
| BootstrapFewShot | Builds few-shot examples; DSPy suggests it for about 10 examples | DSPy docs |
| BootstrapFewShotWithRandomSearch | Adds random search; suggested at 50 or more examples | DSPy docs |
| MIPROv2 | Generates instructions and few-shot examples and uses Bayesian optimization; suggested for runs of 40+ trials with about 200+ examples | DSPy docs, MIPRO paper |
| GEPA | Reflects in natural language on sampled trajectories to diagnose problems and propose prompt updates, keeping a Pareto frontier of candidates | GEPA paper |
| BootstrapFinetune | Fine-tunes weights instead of the prompt | DSPy docs |
DSPy also lists LabeledFewShot, COPRO, SIMBA, KNNFewShot, Ensemble and BetterTogether.
The papers report gains, with the caveat that these are the authors’ numbers on their chosen tasks:
- The MIPRO paper says it outperforms baseline optimizers on five of seven multi-stage programs with Llama-3-8B, with accuracy gains up to 13%.
- The GEPA authors report beating GRPO, a reinforcement learning method, by 6% on average and up to 20% with up to 35x fewer rollouts across six tasks, and beating MIPROv2 by over 10%. The paper was accepted as an ICLR 2026 oral.
Cost and risk
DSPy says a typical simple optimization run costs on the order of $2 and takes around ten minutes. Runs can cost anywhere from a few cents to tens of dollars, and it warns to be careful with very large models or datasets.
The larger risk is the metric. An optimizer pushes the score up, so a weak metric produces a prompt that is good at the metric and nothing else. OpenAI makes a related point: an optimized prompt can perform worse than your original on specific inputs, so review and test them before production. Hold out some examples the optimizer never saw and compare before and after on those.
How to run a prompt optimization
- Write the metric first, and check it on a few outputs you already know are good and bad. See LLM evaluation.
- Collect inputs. Real failures make better test cases than invented ones.
- Run your current prompt to get a baseline score.
- Try a rewriter or an optimizer, or edit by hand.
- Score the new prompt on held-out inputs, not the ones used to optimize.
- Ship only if it wins, and keep the old version so you can roll back.
Optimizing wording and examples has limits. If the model lacks the information, fix what you send it, which is context engineering, or try a different model.
Running the comparison in HoneyHive
HoneyHive’s docs describe experiments as a way to compare prompts, models or configurations side by side to see which changes improve performance. You can build datasets from failing production traces and score runs with LLM or code evaluators. That fits steps 3 and 5 above, whichever tool produced the candidate prompt. Prompt management versions and deploys the winner.
FAQ
What is automatic prompt optimization?
A program searches for a better prompt, scoring each candidate with a metric you define. DSPy describes its optimizers as tuning "the prompts and/or the LM weights" of a program so a metric goes up.
What do I need to optimize a prompt automatically?
DSPy lists three inputs. A program, a metric that scores the program's output, and training inputs, which can be as few as 5 or 10 examples without labels.
Can an optimized prompt get worse?
Yes. OpenAI warns that an optimized prompt can perform worse than your original on specific inputs, so test it before production.
Is OpenAI's dataset-backed prompt optimizer still available?
OpenAI's docs say it is being deprecated with the Evals platform, with Evals read-only for existing users in October 2026 and shutdown scheduled for the end of November 2026. Check its deprecations page for the current timeline.