mlsanityData to 7 Oct 2026

Measuring Progress on Scalable Oversight for Large Language Models

arXiv 2211.03540 ↗First posted 4 Nov 2022cs.HC cs.AI cs.CLcited by 212

Not on the map. Why?

Cited by 47 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

23 of 31 references.

References · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
11 more
  1. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  2. 2018AI safety via debatecited by 384
  3. 2022Self-critiquing models for assisting human evaluatorscited by 372
  4. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  5. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  6. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  7. 2018Supervising strong learners by amplifying weak expertscited by 139
  8. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
  9. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  10. 2022Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questionscited by 22
  11. 2022Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questionscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 34; 2024: 54; 2025: 53; 2026: 68.

212 papers.

Sort:

By area:

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  8. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  9. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  10. 2023Llemma: An Open Language Model For Mathematicscited by 385
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  12. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
38 more
  1. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  2. 2023Large Language Model Alignment: A Surveycited by 241
  3. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  4. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  5. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  7. 2024A Roadmap to Pluralistic Alignmentcited by 175
  8. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  9. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  10. 2024LLM Critics Help Catch LLM Bugscited by 138
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  12. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  13. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  14. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  15. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  16. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  17. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  18. 2022Evaluating Human-Language Model Interactioncited by 90
  19. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  20. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  21. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  22. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  23. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  24. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  25. 2024Can Language Models Learn to Skip Steps?cited by 59
  26. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  27. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  28. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  29. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  30. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  31. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  32. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  33. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  34. 2023Testing Language Model Agents Safely in the Wildcited by 45
  35. 2023Specific versus General Principles for Constitutional AIcited by 41
  36. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  37. 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 40
  38. 2023Debate Helps Supervise Unreliable Expertscited by 39

50 most cited of 212 All 212 (JSON)