mlsanityData to 14 Oct 2026

Robust Critics: Defending LLMs Against Multi-Turn Attacks

arXiv 2607.20472 ↗First posted 24 May 2026cs.AIcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

When a user asks a language model something harmful, is it a genuine attack or a misunderstood but well-meaning question? This ambiguity is one of the central challenges of LLM safety. A model that assumes the worst harms legitimate users; one that assumes the best is easily exploited. The problem is compounded in multi-turn dialogue, where an attacker's true intent may only reveal itself gradually across many exchanges, yet existing safety frameworks apply a contextual bandit treatment, ignoring the trajectory of the conversation. To that end, we propose Dialogue Critic Guided Sampling (DCGS), a framework that addresses this by inferring user intent at every turn of dialogue. Instead of applying a fixed rule about what is or is not safe, DCGS learns what the user's intent is likely to be based on the full conversational history and generates responses accordingly. Formally, we model adversarial dialogue as a Markov Decision Process and learn value and regret-based critics at both the individual token and utterance (full response) levels, scoring candidate responses via an action-value critic. We prove that this inference-time reweighting approximates exponential tilting of the base policy, guaranteeing improvement in expected return for any finite candidate pool, a property that group-relative objectives do not exhibit. Evaluated on CARES-18k, WildJailbreak, Redbench, and Harmbench, DCGS outperforms strong robust baselines and frontier models on adversarial dialogue tasks. DCGS also transfers to frontier models, improving their robustness without fine-tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 52 of its 62 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

52 of 62 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinement10 Feb
  2. 2025Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning6 Jul
  3. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning12 May
  4. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety6 Mar
  5. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering21 Aug
  6. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  7. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  8. 2024On Prompt-Driven Safeguarding for Large Language Models31 Jan
  9. 2023Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack12 Dec

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
31 more
  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  2. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  3. 2017Robust Adversarial Reinforcement Learningcited by 451
  4. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  7. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 222
  8. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  9. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  10. 2023Controlled Decoding from Language Modelscited by 135
  11. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 125
  12. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  13. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  14. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  15. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  16. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  17. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
  18. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  19. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 36
  20. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
  21. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  22. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  23. 2024Inference-Time Language Model Alignment via Integrated Value Guidancecited by 19
  24. 2025Speculative Safety-Aware Decodingcited by 7
  25. 2025Adversarial Suffix Filtering: a Defense Pipeline for LLMscited by 5
  26. 2025Automatic LLM Red Teamingcited by 5
  27. 2023Regret-Based Defense in Adversarial Reinforcement Learningcited by 3
  28. 2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3
  29. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Modelscited by 3
  30. 2026The Anatomy of Conversational Scams: A Topic-Based Red Teaming Analysis of Multi-Turn Interactions in LLMscited by 2
  31. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent
  2. 2026Stateful Guardrails for Multi-Turn LLM Systems: A Conversational Risk Accumulation Framework
  3. 2026Evaluating Language Model Safety Across Long Adversarial Conversations
  4. 2024Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue
  5. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment
  6. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks
  7. 2026TRACE: Trajectory Aware Reasoning for Multi-Turn Adversarial Conversation Evaluation
  8. 2026Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures