mlsanityData to 7 Oct 2026

MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming

arXiv 2505.17147 ↗First posted 22 May 2025cs.CR cs.AIcited by 31

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The proliferation of jailbreak attacks against large language models (LLMs) highlights the need for robust security measures. However, in multi-round dialogues, malicious intentions may be hidden in interactions, leading LLMs to be more prone to produce harmful responses. In this paper, we propose the Multi-Turn Safety Alignment (ourapproach) framework, to address the challenge of securing LLMs in multi-round interactions. It consists of two stages: In the thought-guided attack learning stage, the red-team model learns about thought-guided multi-round jailbreak attacks to generate adversarial prompts. In the adversarial iterative optimization stage, the red-team model and the target model continuously improve their respective capabilities in interaction. Furthermore, we introduce a multi-turn reinforcement learning algorithm based on future rewards to enhance the robustness of safety alignment. Experimental results show that the red-team model exhibits state-of-the-art attack capabilities, while the target model significantly improves its performance on safety benchmarks.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 52 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

39 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  2. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  12. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
20 more
  1. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  2. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  3. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  4. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  5. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 73
  6. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  7. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialoguecited by 64
  8. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  9. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  10. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 40
  11. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  12. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialoguecited by 28
  13. 2023Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignmentcited by 27
  14. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  15. 2023A Two-Stage Adaptation of Large Language Models for Text Rankingcited by 17
  16. 2024Knowledge Fusion By Evolving Weights of Language Modelscited by 16
  17. 2021A Simple but Effective Bidirectional Framework for Relational Triple Extractioncited by 14
  18. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  19. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  20. 2023Topic-Selective Graph Network for Topic-Focused Summarizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 22.

31 papers.

Sort:

By area:

  1. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 20
  2. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  3. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  4. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  5. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  6. 2026DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMscited by 8
  7. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 6
  8. 2025STAC: When Innocent Tools Form Dangerous Chains for LLM Agentscited by 5
  9. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  10. 2026Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counselingcited by 4
  11. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Rewardcited by 3
  12. 2025Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreakcited by 2
19 more
  1. 2026Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimizationcited by 2
  2. 2026Investigating and Alleviating Harm Amplification in LLM Interactionscited by 2
  3. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  4. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 2
  5. 2026JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Modelscited by 1
  6. 2026Evaluating Temporal Consistency in Multi-Turn Language Modelscited by 1
  7. 2026Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreakingcited by 1
  8. 2026PsychJail: Exploring Psychological Jailbreaks via Multi-Turn Persuasion of LLM Policiescited by 1
  9. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Modelscited by 0
  10. 2025State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Modelscited by 0
  11. 2026RoTRAG: Rule of Thumb Reasoning for Conversation Harm Detection with Retrieval-Augmented Generationcited by 0
  12. 2026One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intentcited by 0
  13. 2026Mitigating Many-shot Jailbreak Attacks with One Single Demonstrationcited by 0
  14. 2026SafeSpec: Fast and Safe LLM via Dynamic Reflective Samplingcited by 0
  15. 2026MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignmentcited by 0
  16. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0
  17. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  18. 2026CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agentscited by 0
  19. 2026Speculative Safety Honeypot: Toward Proactive Defense Against Multi-turn Agent Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Models
  2. 2025Improving LLM Safety Alignment with Dual-Objective Optimization
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  4. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  5. 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning
  6. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  7. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization
  8. 2026AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation