mlsanityData to 7 Oct 2026

Training Language Models to Win Debates with Self-Play Improves Judge Accuracy

arXiv 2409.16636 ↗First posted 25 Sep 2024cs.CL cs.AIcited by 16

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 56 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

37 of 56 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
25 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2018AI safety via debatecited by 384
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  5. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  6. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  7. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  8. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  9. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  10. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  11. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  12. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  13. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  14. 2023Scalable AI Safety via Doubly-Efficient Debatecited by 54
  15. 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debatecited by 48
  16. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  17. 2022Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questionscited by 22
  18. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  19. 2024Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLMcited by 21
  20. 2024Can Language Models Recognize Convincing Arguments?cited by 21
  21. 2024Multi-role Consensus through LLMs Discussions for Vulnerability Detectioncited by 19
  22. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  23. 2022Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questionscited by 16
  24. 2024Combating Adversarial Attacks with Multi-Agent Debatecited by 15
  25. 2024Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like Onecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 17
  2. 2025An alignment safety case sketch based on debatecited by 12
  3. 2025AI Agent Behavioral Sciencecited by 5
  4. 2025AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefscited by 4
  5. 2025A Benchmark for Scalable Oversight Protocolscited by 2
  6. 2025Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalizationcited by 1
  7. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 1
  8. 2025Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RLcited by 1
  9. 2025Self-evolving expertise in complex non-verifiable subject domains: dialogue as implicit meta-RLcited by 1
  10. 2026Debate Training Reduces Reward Hacking in RLAIFcited by 1
  11. 2025Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategiescited by 0
  12. 2026RuleSmith: Multi-Agent LLMs for Automated Game Balancingcited by 0
4 more
  1. 2026Towards Robust Argumentative Essay Understanding via TIDE: An Interactive Framework with Trial and Debatecited by 0
  2. 2026Collaborative Disagreement Resolution for Scalable Oversightcited by 0
  3. 2026CoEvo: Oracle-Grounded Self-Evolution of a Single Model for Multi-Step Causal Reasoningcited by 0
  4. 2026Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate
  2. 2025When Two LLMs Debate, Both Think They'll Win
  3. 2025DEBATE, TRAIN, EVOLVE: Self Evolution of Language Model Reasoning
  4. 2025When to Trust Context: Self-Reflective Debates for Context Reliability
  5. 2025DS@GT at Touch'e: Large Language Models for Retrieval-Augmented Debate
  6. 2025DebateBench: A Challenging Long Context Reasoning Benchmark For Large Language Models
  7. 2026AI Safety via Debate is Compromised by Cognitive Biases
  8. 2025Debating for Better Reasoning: An Unsupervised Multimodal Approach