mlsanityData to 7 Oct 2026

Scalable Oversight for Superhuman AI via Recursive Self-Critiquing

arXiv 2502.04675 ↗First posted 7 Feb 2025cs.AI cs.CL cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As AI capabilities increasingly surpass human proficiency in complex tasks, current alignment techniques, including SFT and RLHF, face fundamental challenges in ensuring reliable oversight. These methods rely on direct human assessment and become impractical when AI outputs exceed human cognitive thresholds. In response to this challenge, we explore two hypotheses: (1) textitCritique of critique can be easier than critique itself, extending the widely-accepted observation that verification is easier than generation to the critique domain, as critique itself is a specialized form of generation; (2) textitThis difficulty relationship holds recursively, suggesting that when direct evaluation is infeasible, performing higher-order critiques (e.g., critique of critique of critique) offers a more tractable supervision pathway. We conduct Human-Human, Human-AI, and AI-AI experiments to investigate the potential of recursive self-critiquing for AI supervision. Our results highlight recursive critique as a promising approach for scalable AI oversight.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 44 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

34 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LLM Critics Help Catch LLM Bugs28 Jun
  2. 2024Debating with More Persuasive LLMs Leads to More Truthful Answers9 Feb
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
18 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2018AI safety via debatecited by 384
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  8. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  9. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  10. 2018Supervising strong learners by amplifying weak expertscited by 139
  11. 2018Categorizing Variants of Goodhart's Lawcited by 132
  12. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  13. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  14. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  15. 2024A Survey on Deep Learning for Theorem Provingcited by 55
  16. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  17. 2023Debate Helps Supervise Unreliable Expertscited by 39
  18. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  2. 2025Distributional AGI Safetycited by 8
  3. 2026Test-time Recursive Thinking: Self-Improvement without External Feedbackcited by 7
  4. 2025AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Scalable Oversight via Partitioned Human Supervision
  2. 2022Self-critiquing models for assisting human evaluators
  3. 2026Self-Evaluating Recursive Agents
  4. 2026Rationalize: Shared Semantic Reasoning for Human-AI Alignment
  5. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  6. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment
  7. 2026Automated Benchmark Auditing for AI Agents and Large Language Models
  8. 2025Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgment