How to manage prompts in production

Managing prompts in production means deciding where each prompt lives, recording what produced each output, and releasing changes through a process you can test and undo. A prompt is code that a model executes, so it needs the same review, versions and rollback as the rest of your application. The concept of versions is covered in prompt versioning; the steps below cover the whole loop.

Decide where prompts live

There are two common homes, and they trade off differently.

OpenAI’s guide recommends the first option for its API: “Store production prompts in your application code instead of creating reusable prompt objects.” It says prompt creation is de-emphasized beginning June 3, 2026, and that v1/prompts is scheduled to shut down on November 30, 2026. If your integration calls a saved prompt by ID or version, the guide points to a prompt object migration guide. Moving a provider-hosted prompt store is a cost you can avoid by keeping an exportable copy of every prompt, wherever it lives.

For staged releases, the same guide suggests rolling out changes through your deployment system, using feature flags or configuration. Either home works with that.

What to manage with each prompt

A prompt is more than its text. The behavior you see comes from the template, the model and the settings together.

• The template and its variables. HoneyHive’s docs describe a prompt as a chat template with {{variable_name}} placeholders that are replaced with real values when you fetch it in code. OpenAI’s guide says to use typed function arguments or schemas for dynamic values such as customer data, files or task options.
• The model and its settings. The HoneyHive prompt also carries a provider, a model and hyperparameters such as temperature, max tokens and top P. A prompt that works on one model can behave differently on another. OpenAI advises pinning production applications to specific model snapshots, because even snapshots within one model family can differ.
• A name and an environment. HoneyHive’s docs define three environments per project, dev, staging and prod, and you deploy a saved prompt to one of them. See prompt versioning for how to name and number versions.

Release a change safely

A change to a prompt can fix one case and break another, so treat each one as a release.

1. Fork or branch first. In HoneyHive, saving over an existing configuration overwrites it, so the docs recommend forking to keep a working version before you experiment. In code, a branch does the same job.
2. Test it against a fixed set of cases. OpenAI’s guide recommends building tests and evaluation suites that measure prompt behavior and adding representative fixtures, tests and evaluation checks before changing production prompts. A golden dataset is the usual fixed set; LLM evaluation covers the metrics.
3. Gate the merge on the result. HoneyHive’s CI guide shows a GitHub Actions workflow that runs the evaluation for the pull request and for the base commit, compares them, and fails the check when any metric degrades, which the guide calls the minimal threshold. Improved metrics do not block the pull request.
4. Deploy to staging, then prod. Promote through environments instead of editing production directly.
5. Keep the previous version ready. Rolling back is a redeploy or a revert only if the old version still exists. The versioning page covers how rollback works when versions are never overwritten.

For improving a prompt systematically rather than by hand, see prompt optimization.

Fetch prompts without slowing the app

If prompts live in a store, your application reads them at runtime. HoneyHive’s guide to using prompts in code lists the configurations with the Python SDK and filters them by name and env, and its example wraps the lookup in functools.lru_cache. The guide says to call cache_clear() when prompts change and recommends adding a TTL so cached prompts refresh on their own. It also shows exporting a deployed configuration to YAML, which gives you a file copy you can keep in your repository.

Decide in advance what the application does when the fetch fails. A cached or exported copy is a safe fallback; an error on every request is not.

Tie every output to the prompt that made it

A version number does nothing for you if traces do not carry it. HoneyHive’s tracing docs list config keys you can log on a session or a span, including prompt_name, prompt_version, prompt_id, model, temperature and ab_variant, and they separate config, which controls generation, from metadata, which holds request context such as a request ID. With those on every call you can filter production traces by prompt version, find the traces a change affected and compare two versions. Online experiments tag traces with experiment IDs and variant names for A/B comparisons. For the trace side, see agent tracing.

Managing prompts in HoneyHive

In HoneyHive you write a prompt in the Playground under Studio, run it against sample inputs, save it as a configuration and deploy it to dev, staging or prod from Studio > Prompts. Any traced production call can be opened in the Playground with its template, model and hyperparameters loaded. Your application then fetches the deployed prompt by name and environment with the SDK. The overview page also covers sharing a saved prompt with teammates, who can view and fork it.

FAQ

What is prompt management?

Prompt management is how a team stores, versions, releases and traces the prompts an application sends to a model. It covers where the prompt text lives, who can change it, how a change reaches production and how you tell which prompt produced a given output. For the versioning part alone, see prompt versioning.

Should prompts live in code or in a separate store?

Either can work. OpenAI recommends storing production prompts in application code and is deprecating its reusable prompt objects. A separate store such as the HoneyHive prompt registry lets people change a prompt without a code release, at the cost of a runtime fetch you need to cache and test.

What should I track alongside a prompt?

The model and provider, the hyperparameters and the variables the template expects. HoneyHive's tracing docs list prompt_name, prompt_version, model, temperature and similar keys for a trace's config.

How do I stop a prompt change from breaking production?

Run an evaluation before the change ships and compare it with the current prompt. OpenAI advises adding representative fixtures, tests and evaluation checks before changing production prompts. See LLM evaluation.

Do I need to change the model when I change the prompt?

Not necessarily, but treat a model change as a prompt change. OpenAI notes that even different snapshots within one model family can produce different results, so test the pair.