mlsanityData to 7 Oct 2026

AI safety via debate

arXiv 1805.00899 ↗First posted 2 May 2018stat.ML cs.LGcited by 384

Not on the map. Why?

Cited by 100 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human to directly judge. To help address this concern, we propose training agents via self play on a zero sum debate game. Given a question or proposed action, two agents take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information. In an analogy to complexity theory, debate with optimal play can answer any question in PSPACE given polynomial time judges (direct judging answers only NP questions). In practice, whether debate works involves empirical questions about humans and the tasks we want AIs to perform, plus theoretical questions about the meaning of AI alignment. We report results on an initial MNIST experiment where agents compete to convince a sparse classifier, boosting the classifier's accuracy from 59.4% to 88.9% given 6 pixels and from 48.2% to 85.2% given 4 pixels. Finally, we discuss theoretical and practical aspects of the debate model, focusing on potential weaknesses as the model scales up, and we propose future human and computer experiments to test these properties.

Key facts. Cited by 384 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 12; 2022: 22; 2023: 40; 2024: 49; 2025: 109; 2026: 150.

384 papers.

Sort:

By area:

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  7. 2023Cognitive Architectures for Language Agentscited by 422
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2021Unsolved Problems in ML Safetycited by 291
  12. 2022Teaching language models to support answers with verified quotescited by 290
  13. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  14. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  15. 2023Challenges and Applications of Large Language Modelscited by 251
  16. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  17. 2023Large Language Model Alignment: A Surveycited by 241
  18. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  19. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  20. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  21. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  22. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  23. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  24. 2021Alignment of Language Agentscited by 167
  25. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  26. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  27. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  28. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  29. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  30. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  31. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  32. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  33. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  34. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  35. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  36. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  37. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  38. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67

50 most cited of 384 All 384 (JSON)