What is AI red teaming?

AI red teaming is adversarial testing of an AI system: testers try to make it misbehave so the flaws are found before attackers or users find them. IBM defines it as “a way of interactively testing AI models to protect against harmful behavior”, and the Georgetown CSET definition is “an evaluation methodology to discover flaws and vulnerabilities in AI systems.”

It targets two kinds of problem: security weaknesses such as prompt injection, and harmful output such as toxic, biased or inaccurate content.

Where the term comes from

NIST’s glossary defines a red team as “a group of people authorized and organized to emulate a potential adversary’s attack or exploitation capabilities” against an organization’s security posture. The goal is to show the effects of a successful attack and to show defenders (the blue team) what works. NIST’s glossary page has no separate AI entry, so the AI meaning comes from practitioners.

IBM says the IT industry borrowed the idea to probe networks, systems and software for weaknesses attackers could exploit. With generative AI, the scope grows to harms such as hate speech, hallucinated facts, copyrighted material and private data.

What is different about AI

IBM’s Pin-Yu Chen says testing a generative model is “not like a classifier, where you know the outcomes,” so it needs more interactive testing. The same model can answer one phrasing safely and another unsafely, so one pass proves little.

Microsoft’s team, after red teaming more than 100 generative AI products, drew this lesson: “AI red teams should pay attention to cyberattack vectors both old and new.” Testers should look for familiar application security flaws as well as AI-specific ones like prompt injection. See prompt injection for that attack.

Attacks testers try

IBM names several:

• Role-play jailbreaks, where the bot is asked to act as a character that breaks the rules.
• Translating a prompt into a rarely used language.
• Appending AI-generated gibberish to a prompt.

Early jailbreaks were written by hand. IBM says most jailbreak prompts are now generated by “red team” LLMs that “can produce a wider variety of prompts in limitless quantities.”

Human testers and automation

Part of the work	Who does it, per Microsoft
Producing prompts, orchestrating attacks, scoring results	Automation helps with all three
Subject-matter expertise, cultural context, judgment on emotionally sensitive interactions	People

Source: Microsoft. Its tip is to “adopt tools like PyRIT to scale up operations but keep humans in the red teaming loop.” PyRIT is an open-source toolkit for finding vulnerabilities in your own AI systems. IBM agrees that red teaming “is becoming increasingly automated” but that humans “will continue to play an integral role.”

How to scope an exercise

CSET argues the threat model is “the core design feature of a red-teaming exercise that influences all other design choices.” It describes the system, the relevant vulnerabilities and the contexts where they might arise. Its other points about tools:

• Benchmark-style tools are convenient but carry a fixed threat model and may miss dynamic behavior such as multi-turn interaction.
• Evaluation harnesses are more flexible but take more effort and expertise.
• Aggregated scores such as attack success rate compress the variation between prompts and responses. Using generative AI as a scorer “introduces reliability risks,” and the author found no reliability guarantees in the surveyed tools’ documentation.
• A mismatched tool can give “misleading results” and a “false sense of security.”

The OWASP GenAI Red Teaming Guide (version 1.0, 22 January 2025) takes a holistic view across “model evaluation, implementation testing, infrastructure assessment, runtime behavior analysis.” Microsoft publishes its own AI red team guidance, including a threat modeling guide, a taxonomy of machine learning failures and a bug bar for triaging attacks.

Limits

• No settled standard. CSET calls the field “still severely lacking in theoretical grounding and technical best practices” (CSET, October 2025).
• A clean result is not a guarantee. Microsoft says “mitigations do not eliminate risk entirely,” and describes repeated break-fix cycles of testing, measuring and mitigating as the way to raise attackers’ costs.

Red teaming and its neighbors

Red teaming finds problems, and it differs from a penetration test. Other practices fix or measure them, and they work together. CSET itself frames red teaming as an evaluation method, so it sits alongside LLM evaluation and the lifecycle checks in TEVV. The fixes it prompts often land in AI guardrails.

Re-running attacks against your own app

A red team finding is most useful when you can replay it. In HoneyHive you can turn found attack prompts into a dataset and score the outputs with an evaluator in an offline experiment, to catch regressions before deploy (experiments docs). Online evaluations can score production traces automatically (online evals docs). Tracing makes it easier to see which step failed; see agent observability.

FAQ

What is red teaming in AI?

Adversarial testing of an AI system to find flaws before others do. CSET calls it "an evaluation methodology to discover flaws and vulnerabilities in AI systems", and IBM calls it "a way of interactively testing AI models to protect against harmful behavior."

Can red teaming be fully automated?

Not according to Microsoft, which says red teaming "can't be automated entirely". Automation scales prompts, attack orchestration and scoring, while people bring subject-matter expertise and judgment.

Does red teaming make an AI system safe?

No. Microsoft's team found that "mitigations do not eliminate risk entirely". Repeated rounds of testing and fixing raise the cost of an attack.

Is there a standard way to run an AI red team?

Not yet. CSET says there are "few established standards or best practices". The OWASP GenAI Red Teaming Guide is one published structure.