# X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents

arXiv: https://arxiv.org/abs/2504.13203 · HTML: https://mlsanity.com/paper/2504.13203/

## Key facts

- Cited by 125 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 44 of its 70 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 44)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks](https://mlsanity.com/paper/2403.04783/) (2024)
- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://mlsanity.com/paper/2406.13352/) (2024)

## Built on by (most cited first, 30 of 125)

- [Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models](https://mlsanity.com/paper/2506.07468/) (2025)
- [AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks](https://mlsanity.com/paper/2602.16901/) (2026)
- [ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents](https://mlsanity.com/paper/2509.22830/) (2025)
- [Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents](https://mlsanity.com/paper/2602.13379/) (2026)
- [Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection](https://mlsanity.com/paper/2507.02844/) (2025)
- [Reasoning as an Adaptive Defense for Safety](https://mlsanity.com/paper/2507.00971/) (2025)
- [SoK: Evaluating Jailbreak Guardrails for Large Language Models](https://mlsanity.com/paper/2506.10597/) (2025)
- [OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs](https://mlsanity.com/paper/2601.01592/) (2026)
- [MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety](https://mlsanity.com/paper/2602.01539/) (2026)
- [SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning](https://mlsanity.com/paper/2505.16186/) (2025)
- [WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance](https://mlsanity.com/paper/2511.12997/) (2025)
- [TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards](https://mlsanity.com/paper/2512.07761/) (2025)
- [CoP: Agentic Red-teaming for Large Language Models using Composition of Principles](https://mlsanity.com/paper/2506.00781/) (2025)
- [Large Reasoning Models Are Autonomous Jailbreak Agents](https://mlsanity.com/paper/2508.04039/) (2025)
- [Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD](https://mlsanity.com/paper/2508.17450/) (2025)
- [PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agents](https://mlsanity.com/paper/2509.23614/) (2025)
- [SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attacks](https://mlsanity.com/paper/2602.06854/) (2026)
- [Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models](https://mlsanity.com/paper/2507.05248/) (2025)
- [Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studies](https://mlsanity.com/paper/2510.14312/) (2025)
- [ProbeLLM: Automating Principled Diagnosis of LLM Failures](https://mlsanity.com/paper/2602.12966/) (2026)
- [Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systems](https://mlsanity.com/paper/2603.00131/) (2026)
- [STAC: When Innocent Tools Form Dangerous Chains for LLM Agents](https://mlsanity.com/paper/2509.25624/) (2025)
- [Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks](https://mlsanity.com/paper/2510.02286/) (2025)
- [Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs](https://mlsanity.com/paper/2511.12710/) (2025)
- [DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMs](https://mlsanity.com/paper/2602.16935/) (2026)
- [MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety](https://mlsanity.com/paper/2605.01687/) (2026)
- [SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues](https://mlsanity.com/paper/2506.00668/) (2025)
- [Proactive defense against LLM Jailbreak](https://mlsanity.com/paper/2510.05052/) (2025)
- [Automating Deception: Scalable Multi-Turn LLM Jailbreaks](https://mlsanity.com/paper/2511.19517/) (2025)
- [The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search](https://mlsanity.com/paper/2512.01353/) (2025)

## Abstract (arXiv)

Multi-turn interactions with language models (LMs) pose critical safety risks, as harmful intent can be strategically spread across exchanges. Yet, the vast majority of prior work has focused on single-turn safety, while adaptability and diversity remain among the key challenges of multi-turn red-teaming. To address these challenges, we present X-Teaming, a scalable framework that systematically explores how seemingly harmless interactions escalate into harmful outcomes and generates corresponding attack scenarios. X-Teaming employs collaborative agents for planning, attack optimization, and verification, achieving state-of-the-art multi-turn jailbreak effectiveness and diversity with success rates up to 98.1% across representative leading open-weight and closed-source models. In particular, X-Teaming achieves a 96.2% attack success rate against the latest Claude 3.7 Sonnet model, which has been considered nearly immune to single-turn attacks. Building on X-Teaming, we introduce XGuard-Train, an open-source multi-turn safety training dataset that is 20x larger than the previous best resource, comprising 30K interactive jailbreaks, designed to enable robust multi-turn safety alignment for LMs. Our work offers essential tools and insights for mitigating sophisticated conversational attacks, advancing the multi-turn safety of LMs.