mlsanityData to 14 Oct 2026

On scalable oversight with weak LLMs judging strong LLMs

arXiv 2407.04622 ↗First posted 5 Jul 2024cs.LGcited by 99

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a judge; consultancy, where a single AI tries to convince a judge that asks questions; and compare to a baseline of direct question-answering, where the judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include mathematics, coding, logic and multimodal reasoning asymmetries. We find that debate outperforms consultancy across all tasks when the consultant is randomly assigned to argue for the correct/incorrect answer. Comparing debate to direct question answering, the results depend on the type of task: in extractive QA tasks with information asymmetry debate outperforms direct question answering, but in other tasks without information asymmetry the results are mixed. Previous work assigned debaters/consultants an answer to argue for. When we allow them to instead choose which answer to argue for, we find judges are less frequently convinced by the wrong answer in debate than in consultancy. Further, we find that stronger debater models increase judge accuracy, though more modestly than in previous studies.

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). Builds on 41 of its 71 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

41 of 71 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  12. 2016Concrete Problems in AI Safetycited by 1,642
29 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  4. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  5. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  6. 2022PAL: Program-aided Language Modelscited by 652
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  8. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  9. 2018AI safety via debatecited by 400
  10. 2022Self-critiquing models for assisting human evaluatorscited by 380
  11. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  12. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  13. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  14. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  15. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  16. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  17. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  18. 2024Many-Shot In-Context Learningcited by 206
  19. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  20. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  21. 2018Supervising strong learners by amplifying weak expertscited by 145
  22. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  23. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  24. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  25. 2023Scalable AI Safety via Doubly-Efficient Debatecited by 57
  26. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddingscited by 51
  27. 2023Debate Helps Supervise Unreliable Expertscited by 43
  28. 2022Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questionscited by 23
  29. 2022Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 43; 2026: 47.

99 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 66
  3. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 55
  4. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  5. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  6. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
  7. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  8. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
  9. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  10. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  11. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  12. 2026How AI Impacts Skill Formationcited by 24
38 more
  1. 2025Scaling Laws For Scalable Oversightcited by 23
  2. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 22
  3. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  4. 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 21
  5. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 18
  6. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  7. 2025An alignment safety case sketch based on debatecited by 16
  8. 2024Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agentcited by 14
  9. 2025Avoiding Obfuscation with Prover-Estimator Debatecited by 14
  10. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 13
  11. 2025When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)cited by 12
  12. 2025Distributional AGI Safetycited by 12
  13. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoningcited by 11
  14. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  15. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  16. 2025Control Tax: The Price of Keeping AI in Checkcited by 10
  17. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 10
  18. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  19. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  20. 2026Training AI Scientists to Replicate Researchcited by 9
  21. 2025A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety Riskscited by 8
  22. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 8
  23. 2025Verdict: A Library for Scaling Judge-Time Computecited by 7
  24. 2025Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creationcited by 7
  25. 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbationscited by 7
  26. 2026Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversightcited by 6
  27. 2025Iterative Label Refinement Matters More than Preference Optimization under Weak Supervisioncited by 5
  28. 2025Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliancecited by 5
  29. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  30. 2026Dynamic Role Assignment for Multi-Agent Debatecited by 5
  31. 2026ReasonOps: Operator Segmentation for LLM Reasoning Tracescited by 5
  32. 2025Scalable Oversight for Superhuman AI via Recursive Self-Critiquingcited by 4
  33. 2025AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefscited by 4
  34. 2025When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?cited by 4
  35. 2025Weak-to-Strong Generalization under Distribution Shiftscited by 4
  36. 2026Debate Training Reduces Reward Hacking in RLAIFcited by 4
  37. 2025FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Researchcited by 3
  38. 2025A Benchmark for Scalable Oversight Protocolscited by 3

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Knowledge Divergence and the Value of Debate for Scalable Oversight
  2. 2026A Theory of Post-hoc Debate Judgement
  3. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  4. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
  5. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  6. 2026JuryFlow: Disagreement-Guided Human-in-the-Loop Multi-Agent Evaluation
  7. 2024Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debate
  8. 2024Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation