mlsanityData to 14 Oct 2026

X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents

arXiv 2504.13203 ↗First posted 15 Apr 2025cs.CR cs.AI cs.CLcited by 125

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Multi-turn interactions with language models (LMs) pose critical safety risks, as harmful intent can be strategically spread across exchanges. Yet, the vast majority of prior work has focused on single-turn safety, while adaptability and diversity remain among the key challenges of multi-turn red-teaming. To address these challenges, we present X-Teaming, a scalable framework that systematically explores how seemingly harmless interactions escalate into harmful outcomes and generates corresponding attack scenarios. X-Teaming employs collaborative agents for planning, attack optimization, and verification, achieving state-of-the-art multi-turn jailbreak effectiveness and diversity with success rates up to 98.1% across representative leading open-weight and closed-source models. In particular, X-Teaming achieves a 96.2% attack success rate against the latest Claude 3.7 Sonnet model, which has been considered nearly immune to single-turn attacks. Building on X-Teaming, we introduce XGuard-Train, an open-source multi-turn safety training dataset that is 20x larger than the previous best resource, comprising 30K interactive jailbreaks, designed to enable robust multi-turn safety alignment for LMs. Our work offers essential tools and insights for mitigating sophisticated conversational attacks, advancing the multi-turn safety of LMs.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 44 of its 70 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

44 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Guardians of the Agentic System: Preventing Many Shots Jailbreak with Agentic System23 Feb
  2. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue6 Nov
  3. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  4. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar

Other references · 40

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
28 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  4. 2023Large Language Models as Optimizerscited by 864
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  6. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  7. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  8. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  9. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  10. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  11. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  12. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  13. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  14. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  15. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  16. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  17. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 127
  18. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  19. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 64
  20. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 56
  21. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  22. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  23. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 44
  24. 2024Are Large Language Models Good Prompt Optimizers?cited by 37
  25. 2024Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizerscited by 35
  26. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  27. 2024Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimizationcited by 18
  28. 2024STAND-Guard: A Small Task-Adaptive Content Moderation Modelcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 37; 2026: 88.

125 papers.

Sort:

By area:

  1. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
  2. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 39
  3. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 31
  4. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 30
  5. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 25
  6. 2025Reasoning as an Adaptive Defense for Safetycited by 23
  7. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 21
  8. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 21
  9. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  10. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  11. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
  12. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 14
38 more
  1. 2025CoP: Agentic Red-teaming for Large Language Models using Composition of Principlescited by 13
  2. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 13
  3. 2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDcited by 13
  4. 2025PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agentscited by 12
  5. 2026SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attackscited by 12
  6. 2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Modelscited by 10
  7. 2025Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studiescited by 10
  8. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 9
  9. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  10. 2025STAC: When Innocent Tools Form Dangerous Chains for LLM Agentscited by 8
  11. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  12. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 8
  13. 2026DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMscited by 8
  14. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safetycited by 8
  15. 2025SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialoguescited by 7
  16. 2025Proactive defense against LLM Jailbreakcited by 7
  17. 2025Automating Deception: Scalable Multi-Turn LLM Jailbreakscited by 7
  18. 2025The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Searchcited by 7
  19. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 7
  20. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 7
  21. 2026TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreakingcited by 7
  22. 2025Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domaincited by 6
  23. 2025When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injectioncited by 6
  24. 2026ICON: Intent-Context Coupling for Efficient Multi-Turn Jailbreak Attackcited by 6
  25. 2025ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attackscited by 5
  26. 2025Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energycited by 5
  27. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  28. 2026AgenticRed: Evolving Agentic Systems for Red-Teamingcited by 5
  29. 2025Pattern Enhanced Multi-Turn Jailbreaking: Exploiting Structural Vulnerabilities in Large Language Modelscited by 4
  30. 2025JPRO: Automated Multimodal Jailbreaking via Multi-Agent Collaboration Frameworkcited by 4
  31. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  32. 2026State-Dependent Safety Failures in Multi-Turn Language Model Interactioncited by 4
  33. 2026Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planningcited by 4
  34. 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploitscited by 3
  35. 2025Agent-based simulation of online social networks and disinformationcited by 3
  36. 2026RvB: Automating AI System Hardening via Iterative Red-Blue Gamescited by 3
  37. 2026RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Modelscited by 3
  38. 2026Do Reasoning LLMs Refuse What They Infer in Long Contexts?cited by 3

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking
  2. 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning
  3. 2025Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning
  4. 2025GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication
  5. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
  6. 2025Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors
  7. 2025Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space
  8. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks