# How to red team an LLM application

> Red team an LLM app in rounds: pick people and layers, probe open-ended, build a harm list, record each finding, report, then replay fixes as tests.

Source: https://www.honeyhive.ai/learn/llm-red-teaming (updated 2026-11-05)

To red team an LLM application, run it as a series of planned rounds. Decide who tests and which layer they hit, let them explore without a script, turn what they find into a list of harms, test against that list in later rounds, and replay every finding after each fix. Microsoft's [planning guide](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) says "advance planning is critical to a productive red teaming exercise."

This page is the procedure. For the definition see [AI red teaming](/learn/ai-red-teaming), and for how it differs from a security test see [red teaming vs penetration testing](/learn/red-teaming-vs-pentesting).

## Before the first prompt

### Decide what you are worried about

Start from a threat model, not from a list of jailbreak prompts. The OWASP GenAI Red Teaming Guide, [published](https://genai.owasp.org/2025/01/22/announcing-the-owasp-gen-ai-red-teaming-guide/) in January 2025, describes a "structured, risk-based methodology" that spans model-level problems such as toxicity and bias and system-level problems such as API misuse and data exposure. Its resource page [names](https://genai.owasp.org/resource/genai-red-teaming-guide/) four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis.

Two references help you name the threats. The [OWASP Top 10 for LLM applications](/learn/owasp-llm-top-10) lists the risk categories, and [MITRE ATLAS](https://github.com/mitre-atlas/atlas-data) is "a public knowledge base of adversary TTPs targeting AI systems," where a technique is the "how" and a case study documents a real instance. Google's AI Red Team [lists](https://blog.google/technology/safety-security/googles-ai-red-team-the-ethical-hackers-making-ai-safer/) the attacks it considers most relevant: prompt attacks, training data extraction, backdooring the model, adversarial examples, data poisoning and exfiltration. Microsoft's team adds that testers should watch for ["cyberattack vectors both old and new"](https://www.microsoft.com/en-us/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/), so old application flaws belong in scope next to [prompt injection](/learn/prompt-injection).

### Choose the testers

Microsoft [recommends](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) a diverse group across AI, social science and security, with domain experts when the product needs them. Its example is a chatbot for health care providers, where medical experts help find domain risks. It also wants two mindsets: testers with adversarial and security experience, who are "essential for understanding security risks", and ordinary users who were not involved in development and who can show harms regular users would meet.

Then assign them. Microsoft's [guide](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) says security specialists can probe for jailbreaks, meta prompt extraction and cyberattack content. Over several rounds you can rotate assignments for fresh perspectives, and later you can assign testers to application features to cover the whole product. Benign-scenario testers may need less time than adversarial ones.

OpenAI's paper on [external red teaming](https://arxiv.org/html/2503.16431) adds that different systems need different cohorts, which can include academic groups, specialist red-teaming organizations and public bounty or challenge arenas.

### Choose the layer

| Layer | How to test it | Why |
|---|---|---|
| Base model with its safety system | Usually through an API endpoint | Shows the risk surface and gaps your application context may expose |
| Your application | Best through the UI, ideally production | Closest to real use |
| Both, before and after mitigations | Same tests, two versions | Shows whether a mitigation works |

Source: Microsoft's [planning guide](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming), which also says to state which endpoints were used when you report results and to retest on the production endpoint if you tested elsewhere. OpenAI [describes](https://arxiv.org/html/2503.16431) the same trade: early pre-mitigation access to assess base risk and find novel risks, later testing for residual risk and how well mitigations hold.

### Brief the testers and set up recording

Give each tester written instructions: the purpose of the round, the product and features in scope and how to reach them, the kinds of issues to look for, focus areas, how much time to spend, how to record results and whom to ask. OpenAI [adds](https://arxiv.org/html/2503.16431) that exploratory rounds get open-ended instructions while structured rounds get pre-set domains and threat models.

Decide what gets recorded before testing starts. Microsoft [suggests](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) the date, a unique ID for the input and output pair "for reproducibility purposes", the input prompt, and a description or screenshot of the output. OpenAI's [list](https://arxiv.org/html/2503.16431) is similar: prompt and generation pairs or whole conversations, a category, a risk level on a defined scale, and a note on how the level was chosen. Microsoft says a shared spreadsheet is often the simplest format, and it lets testers see each other's examples and avoid duplicates.

## Running the rounds

1. **Open-ended testing.** Ask testers to explore and document any problematic content instead of hunting specific harms. Microsoft [says](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) this lets them creatively cover a wide range of issues and uncover blind spots in your picture of the risk surface.
2. **Build a harm list.** Write each harm with a definition and examples, and hand the list to testers in later rounds.
3. **Guided rounds.** Keep probing the known harms and the mitigations for them, and expect new harms to surface. Add them to the list and be ready to shift priorities. Prioritize by severity and by the context in which a harm is likely to appear.
4. **Decide what each finding means.** OpenAI's [paper](https://arxiv.org/html/2503.16431) says teams first work out which examples fall under existing policy and whether they violate it. If no policy covers a finding, decide whether to write one or change the intended behavior.

Run a first manual round before you build systematic measurements or mitigations. Microsoft [calls](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) that a best practice, because the round produces the list of harms that tells you what to measure and fix.

Automation fits inside this loop. Microsoft's team [found](https://www.microsoft.com/en-us/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/) that tools help with prompt creation, attack orchestration and response scoring, while "red teaming can't be automated entirely" and relies heavily on human expertise.

While testing runs, someone should stay on standby for access problems and questions, watch progress in the shared file and send reminders.

## After each round

Microsoft [recommends](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) a short report to stakeholders at a regular interval with the top issues, a link to the raw data, a preview of the next rounds and an acknowledgment of the testers. Add a content warning if the report contains harmful examples.

State plainly that the report identifies risk and does not measure it. Microsoft says people should not read specific examples as a metric for how common a harm is. The measurement step is a separate piece of work: OpenAI's [paper](https://arxiv.org/html/2503.16431) describes using datasets from human red teamers "to create repeatable safety evaluations", with testers supplying seed prompts that classifiers or other evaluators then score. The same idea applies at small scale in [LLM evaluation](/learn/llm-evaluation), and [synthetic test data](/learn/synthetic-test-data-for-evals) can widen a few found attacks into a larger set.

Fixing is its own cycle. Microsoft [describes](https://www.microsoft.com/en-us/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/) break-fix cycles of testing, measuring and mitigating as a way to raise the cost of an attack, and says "mitigations do not eliminate risk entirely." Google's AI Red Team likewise [notes](https://blog.google/technology/safety-security/googles-ai-red-team-the-ethical-hackers-making-ai-safer/) that some attacks have no simple fix, while standard security controls reduce risk and many attacks can be spotted with the detection methods used for conventional attacks. Runtime defenses are covered in [AI guardrails](/learn/ai-guardrails).

## Limits to plan around

OpenAI's [paper](https://arxiv.org/html/2503.16431) lists several. A round is a point-in-time check, so "risks surfaced in one point in time red teaming effort may be under-assessed" as the system changes. Publishing a working jailbreak can speed misuse, so access to findings needs control and responsible disclosure. Testers can be harmed by the content they handle, and the paper recommends mental health resources, fair pay and informed consent. Red teaming "is not a panacea for risk assessment."

Anthropic [wrote](https://www.anthropic.com/news/challenges-in-red-teaming-ai-systems) in June 2024 that the field lacks standardized practices, which makes it hard to compare the safety of different systems, and that turning red team results into lasting value is a challenge in itself. The replay step below addresses that second point.

## Replaying findings in HoneyHive

A finding is only reusable if it can be run again. In HoneyHive, you can [upload attack prompts as a dataset](https://docs.honeyhive.ai/v2/datasets/import.md) or [curate one from production traces](https://docs.honeyhive.ai/v2/datasets/dataset-curation.md), then score outputs with an evaluator in an [offline experiment](https://docs.honeyhive.ai/v2/evaluation/introduction.md) to catch regressions before deploy. [Online evaluations](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) can score production traces automatically, and [agent observability](/learn/agent-observability) shows which step of a trace went wrong.

## FAQ

### How do you red team an LLM application?

Plan who tests and what layer they hit, run open-ended testing, turn the findings into a harm list, test against that list in later rounds, record every input and output, and report after each round. Microsoft's [planning guide](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) lays out these steps.

### Should I test the model or the application?

Both. Microsoft [says](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) to test the base model (usually through an API endpoint), the application (best through its UI), and both before and after mitigations, and to use the production UI as much as possible because it most closely resembles real use.

### Who should do the testing?

A mix. Microsoft [recommends](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) people with adversarial and security experience plus ordinary users who were not involved in building the app, and domain experts where the product needs them.

### Can I automate LLM red teaming?

Partly. Microsoft's team [found](https://www.microsoft.com/en-us/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/) that automation helps with prompt creation, attack orchestration and response scoring, but that red teaming can't be automated entirely.

### Does a red team report tell me how often a harm happens?

No. Microsoft [warns](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming) that people should not read specific examples as a metric for how common a harm is. Measurement comes after red teaming, with a repeatable evaluation.