# Scaling Laws For Scalable Oversight

arXiv: https://arxiv.org/abs/2504.18530 · HTML: https://mlsanity.com/paper/2504.18530/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 33 of its 90 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [Revisiting Neural Scaling Laws in Language and Vision](https://mlsanity.com/paper/2209.06640/) (2022)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [On scalable oversight with weak LLMs judging strong LLMs](https://mlsanity.com/paper/2407.04622/) (2024)
- [Scalable AI Safety via Doubly-Efficient Debate](https://mlsanity.com/paper/2311.14125/) (2023)
- [Theoretical Analysis of Weak-to-Strong Generalization](https://mlsanity.com/paper/2405.16043/) (2024)
- [An Approach to Technical AGI Safety and Security](https://mlsanity.com/paper/2504.01849/) (2025)
- [Debate Helps Supervise Unreliable Experts](https://mlsanity.com/paper/2311.08702/) (2023)

## Built on by (most cited first, 23 of 23)

- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Control Tax: The Price of Keeping AI in Check](https://mlsanity.com/paper/2506.05296/) (2025)
- [Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safety](https://mlsanity.com/paper/2506.12482/) (2025)
- [Training Agents to Self-Report Misbehavior](https://mlsanity.com/paper/2602.22303/) (2026)
- [ReasonOps: Operator Segmentation for LLM Reasoning Traces](https://mlsanity.com/paper/2605.29192/) (2026)
- [Training Deliberative Monitors for Black-Box Scheming Detection](https://mlsanity.com/paper/2605.29601/) (2026)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Scaling Reward Modeling without Human Supervision](https://mlsanity.com/paper/2603.02225/) (2026)
- [Debate Helps Weak Judges Reward Stronger Models](https://mlsanity.com/paper/2605.27483/) (2026)
- [Bootstrapped Monitoring: Leveraging Transparent Reasoning to Oversee Stronger AI Agents](https://mlsanity.com/paper/2606.11998/) (2026)
- [Toward Human-AI Complementarity Across Diverse Tasks](https://mlsanity.com/paper/2605.04070/) (2026)
- [The Open-Box Fallacy: Why AI Deployment Needs a Calibrated Verification Regime](https://mlsanity.com/paper/2605.10601/) (2026)
- [Calibrating Conservatism for Scalable Oversight](https://mlsanity.com/paper/2605.28807/) (2026)
- [Minimal Oversight: Uncertainty-Aware Governance for Delegated AI Systems](https://mlsanity.com/paper/2606.15563/) (2026)
- [Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model](https://mlsanity.com/paper/2608.19216/) (2026)
- [When Honesty is Not Enough in AI Debate](https://mlsanity.com/paper/2609.29189/) (2026)
- [Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure](https://mlsanity.com/paper/2609.30217/) (2026)
- [Contract monitoring: governing AI via separation of powers](https://mlsanity.com/paper/2609.32061/) (2026)
- [When Should a Human Take Back Control? Optimal Delegation under Turbulent AI Risk](https://mlsanity.com/paper/2609.32083/) (2026)
- [Measuring Collapse and Correction in Homogeneous-Panel LLM Debate](https://mlsanity.com/paper/2609.35279/) (2026)
- [PyINE: A Framework for Scalable Elicitation and Oversight via Code Execution](https://mlsanity.com/paper/2610.04737/) (2026)
- [Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements](https://mlsanity.com/paper/2610.08540/) (2026)
- [Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap](https://mlsanity.com/paper/2610.15778/) (2026)

## Abstract (arXiv)

Scalable oversight, the process by which weaker AI systems supervise stronger ones, has been proposed as a key strategy to control future superintelligent systems. However, it is still unclear how scalable oversight itself scales. To address this gap, we propose a framework that quantifies the probability of successful oversight as a function of the capabilities of the overseer and the system being overseen. Specifically, our framework models oversight as a game between capability-mismatched players; the players have oversight-specific Elo scores that are a piecewise-linear function of their general intelligence, with two plateaus corresponding to task incompetence and task saturation. We validate our framework with a modified version of the game Nim and then apply it to four oversight games: Mafia, Debate, Backdoor Code and Wargames. For each game, we find scaling laws that approximate how domain performance depends on general AI system capability. We then build on our findings in a theoretical study of Nested Scalable Oversight (NSO), a process in which trusted models oversee untrusted stronger models, which then become the trusted models in the next step. We identify conditions under which NSO succeeds and derive numerically (and in some cases analytically) the optimal number of oversight levels to maximize the probability of oversight success. We also apply our theory to our four oversight games, where we find that NSO success rates at a general Elo gap of 400 are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code, and 9.4% for Wargames; these rates decline further when overseeing stronger systems.