# What is an LLM jailbreak?

> An LLM jailbreak is an input that gets a model to ignore its safety training and produce content it would refuse. How they work and how to test.

Source: https://www.honeyhive.ai/learn/llm-jailbreak (updated 2026-11-07)

An LLM jailbreak is an input that gets a model to bypass its safety training and produce content it would normally refuse. MITRE ATLAS [describes](https://d3fend.mitre.org/offensive-technique/attack/AML.T0054/) the technique as inducing a model to "ignore, circumvent, or override its safety/alignment behaviors", and notes it can use adversarial prompting or modify model weights or safety mechanisms. The prompting side is what most people mean by the word, and it is the focus here.

## Jailbreak versus prompt injection

The two terms overlap, and sources draw the line differently. Simon Willison [defines](https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/) jailbreaking as attacks that try to subvert the safety filters built into the model itself. In his framing, [prompt injection](/learn/prompt-injection) is about applications that merge untrusted input into a trusted prompt: "if there's no concatenation of trusted and untrusted strings, it's not prompt injection."

OWASP [treats](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) jailbreaking as a form of prompt injection in which the input makes the model disregard its safety protocols entirely, and says the terms are often used interchangeably. A practical split: a jailbreak attacks the model's rules, an injection attacks your application's instructions. An agent with tools can face both, and MITRE [links](https://d3fend.mitre.org/offensive-technique/attack/AML.T0054/) jailbreaks to privilege escalation, where the adversary invokes tools or performs actions for their own purposes.

## Why safety training can be bypassed

Wei, Haghtalab and Steinhardt [studied](https://arxiv.org/abs/2307.02483) why jailbreaks succeed against safety-trained models and proposed two failure modes:

| Failure mode | Meaning |
|---|---|
| Competing objectives | The model's capabilities and its safety goals conflict, so a request can pull it toward complying |
| Mismatched generalization | Safety training does not extend to domains where the model still has capabilities |

Using these ideas, the authors built new attacks against GPT-4 and Claude v1.3 and [reported](https://arxiv.org/abs/2307.02483) that they succeeded on every prompt in a set of unsafe requests from the models' red-teaming evaluations. They concluded that safety mechanisms need to be as sophisticated as the underlying model, and that scaling alone will not fix these failures. The work is from July 2023 and names old models, so read it for the mechanism, not for current success rates.

## Attack families

Research describes a few families. No attack prompts are reproduced here.

| Family | What the research says |
|---|---|
| Hand-written prompts shared online | Shen and colleagues [collected](https://arxiv.org/abs/2308.03825) 1,405 in-the-wild jailbreak prompts from December 2022 to December 2023 and found 131 communities, plus 28 accounts that kept optimizing prompts for over 100 days. Five prompts reached a 0.95 attack success rate on GPT-3.5 and GPT-4, and the earliest persisted online for over 240 days. |
| Automatically generated suffixes | In July 2023 Zou and colleagues [generated](https://arxiv.org/abs/2307.15043) adversarial suffixes by optimizing across multiple prompts and models. They reported the suffixes transferred to black-box, publicly released models, including ChatGPT, Bard and Claude. |
| Many-shot jailbreaking | Anthropic [describes](https://www.anthropic.com/research/many-shot-jailbreaking) filling a long prompt with a fabricated dialogue in which an assistant answers harmful queries. A few examples do not override safety training, a very large number does (tested up to 256), and larger models tend to be more vulnerable. Long context windows are what make it feasible. |

The many-shot result ties jailbreaking to the context window: the same capacity that lets a model use long inputs also gives an attacker room to work. See [context window](/learn/context-window).

## Defenses and what they cost

No single control closes the gap, and the published evidence shows tradeoffs.

| Defense | Evidence |
|---|---|
| Classifiers on inputs and outputs | In February 2025 Anthropic [reported](https://www.anthropic.com/research/constitutional-classifiers) a drop in jailbreak success from 86% to 4.4% across 10,000 synthetic prompts, with a 0.38% rise in refusals of harmless queries and 23.7% more compute. Its January 2026 [successor](https://www.anthropic.com/research/next-generation-constitutional-classifiers) brought the overhead down to about 1% compute (an estimate for Claude Opus 4.0 traffic) and a 0.05% refusal rate on harmless queries over one month of Claude Sonnet 4.5 traffic. |
| Classifying and modifying prompts | For many-shot attacks, Anthropic [found](https://www.anthropic.com/research/many-shot-jailbreaking) this cut attack success from 61% to 2% in one case. |
| Fine-tuning the model to refuse | Anthropic [found](https://www.anthropic.com/research/many-shot-jailbreaking) it only delayed the harmful outputs. |
| Limiting context length | It would prevent many-shot attacks but [removes](https://www.anthropic.com/research/many-shot-jailbreaking) the benefit of long inputs. |
| Application controls | OWASP [lists](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) constrained model behavior, validated output formats, input and output filtering, least privilege, human approval for high-risk actions and regular adversarial testing. |

Anthropic's red-teaming results show how hard a good defense is to judge. In a 2025 prototype bug bounty, [183 participants](https://www.anthropic.com/research/constitutional-classifiers) spent over 3,000 hours and found no universal jailbreak. In a public demo, 339 jailbreakers made over 300,000 interactions and one participant found a universal jailbreak. Anthropic says the classifiers may not stop every universal jailbreak and [recommends](https://www.anthropic.com/research/constitutional-classifiers) complementary defenses. Its 2026 successor [reports](https://www.anthropic.com/research/next-generation-constitutional-classifiers) over 1,700 hours of red-teaming without a universal jailbreak found yet, and says no AI system on the market has perfectly robust defenses. OWASP [adds](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) that preventing jailbreaks takes ongoing updates to model training and safety mechanisms, not only application-level safeguards. For the controls you can add around a model, see [AI guardrails](/learn/ai-guardrails); for the wider threat list, [OWASP LLM Top 10](/learn/owasp-llm-top-10) and [LLM security](/learn/llm-security).

## How to measure resistance

Testing is the part you control. JailbreakBench [offers](https://arxiv.org/abs/2404.01318) an open benchmark with 100 behaviors, a standardized evaluation framework with a defined threat model and scoring functions, a repository of attack artifacts and a public leaderboard. Its authors say earlier benchmarks measured costs and success rates in incomparable ways and were often not reproducible. A later version [adds](https://arxiv.org/abs/2404.01318) an over-refusal dataset, which matters because a defense that blocks harmless requests also costs you.

For your own application, keep one fixed set of behaviors, rerun it after every model, prompt or guardrail change, and track both attack success and false refusals. That is the loop described in [LLM red teaming](/learn/llm-red-teaming) and [AI red teaming](/learn/ai-red-teaming).

## Check for it in your own system

Run the fixed set as an [evaluation](/learn/llm-evaluation) before release, then watch live traffic. In HoneyHive, [online evaluations](https://docs.honeyhive.ai/v2/monitoring/onlineevals.md) score production traces in the background with code or LLM evaluators without slowing the application, and the sampling percentage controls what fraction of matching events get evaluated. Tracing the full conversation also helps with multi-turn and long-context attacks, since the whole prompt is on the trace. More on that in [LLM observability](/learn/llm-observability).

## FAQ

### What is an LLM jailbreak?

An input that causes a model to bypass its safety behavior. MITRE ATLAS [describes](https://d3fend.mitre.org/offensive-technique/attack/AML.T0054/) it as inducing a model to "ignore, circumvent, or override its safety/alignment behaviors" so it produces restricted outputs.

### Is a jailbreak the same as prompt injection?

Related, but not identical. Simon Willison [defines](https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/) jailbreaking as attacks on the safety filters built into the model, while prompt injection targets applications that join untrusted input to a trusted prompt. OWASP [calls](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) jailbreaking a form of prompt injection and notes the terms are often used interchangeably.

### Why do jailbreaks work?

Wei, Haghtalab and Steinhardt [propose](https://arxiv.org/abs/2307.02483) two failure modes of safety training: competing objectives, where capabilities and safety goals conflict, and mismatched generalization, where safety training does not cover domains the model can still handle.

### Can a jailbreak be fully prevented?

No defense is proven to stop every one. Anthropic [says](https://www.anthropic.com/research/constitutional-classifiers) its classifiers may not stop every universal jailbreak and recommends complementary defenses. OWASP [says](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) it is unclear whether any fool-proof prevention for prompt injection exists.

### How do you measure jailbreak resistance?

Run a fixed set of harmful behaviors and score how often the model complies. JailbreakBench [provides](https://arxiv.org/abs/2404.01318) 100 behaviors, a standard evaluation framework and a public leaderboard, plus an over-refusal dataset so you also see what a defense blocks by mistake.