mlsanityData to 7 Oct 2026

Efficient LLM Safety Evaluation through Multi-Agent Debate

arXiv 2511.06396 ↗First posted 9 Nov 2025cs.AI cs.CRcited by 2

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Safety evaluation of large language models (LLMs) increasingly relies on LLM-as-a-judge pipelines, but strong judges can still be expensive to use at scale. We study whether structured multi-agent debate can improve judge reliability while keeping backbone size and cost modest. To do so, we introduce HAJailBench, a human-annotated jailbreak benchmark with 11,100 labeled interactions spanning diverse attack methods and target models, and we pair it with a Multi-Agent Judge framework in which critic, defender, and judge agents debate under a shared safety rubric. On HAJailBench, the framework improves over matched small-model prompt baselines and prior multi-agent judges, while remaining more economical than GPT-4o under the evaluated pricing snapshot. Ablation results further show that a small number of debate rounds is sufficient to capture most of the gain. Together, these results support structured, value-aligned debate as a practical design for scalable LLM safety evaluation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 24 of its 27 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

24 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
  2. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework11 Oct

Other references · 22

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  9. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  11. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
10 more
  1. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  2. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  5. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  6. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  7. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  8. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  9. 2024A Performance Study of LLM-Generated Code on Leetcodecited by 48
  10. 2024Small Language Model Can Self-correctcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  2. 2025Aetheria: A multimodal interpretable content safety framework based on multi-agent debate and collaborationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
  3. 2025LLM-Safety Evaluations Lack Robustness
  4. 2026CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safety
  5. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
  6. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates
  7. 2025SafeLawBench: Towards Safe Alignment of Large Language Models
  8. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models