# A Benchmark for Scalable Oversight Protocols

arXiv: https://arxiv.org/abs/2504.03731 · HTML: https://mlsanity.com/paper/2504.03731/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 13 of 13)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [On scalable oversight with weak LLMs judging strong LLMs](https://mlsanity.com/paper/2407.04622/) (2024)
- [Scalable AI Safety via Doubly-Efficient Debate](https://mlsanity.com/paper/2311.14125/) (2023)
- [Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs](https://mlsanity.com/paper/2404.10160/) (2024)
- [Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning](https://mlsanity.com/paper/2402.00667/) (2024)
- [Training Language Models to Win Debates with Self-Play Improves Judge Accuracy](https://mlsanity.com/paper/2409.16636/) (2024)

## Built on by (most cited first, 3 of 3)

- [Scaling Laws For Scalable Oversight](https://mlsanity.com/paper/2504.18530/) (2025)
- [Training Deliberative Monitors for Black-Box Scheming Detection](https://mlsanity.com/paper/2605.29601/) (2026)
- [Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety](https://mlsanity.com/paper/2605.22643/) (2026)

## Abstract (arXiv)

As AI agents surpass human capabilities, scalable oversight -- the problem of effectively supplying human feedback to potentially superhuman AI models -- becomes increasingly critical to ensure alignment. While numerous scalable oversight protocols have been proposed, they lack a systematic empirical framework to evaluate and compare them. While recent works have tried to empirically study scalable oversight protocols -- particularly Debate -- we argue that the experiments they conduct are not generalizable to other protocols. We introduce the scalable oversight benchmark, a principled framework for evaluating human feedback mechanisms based on our agent score difference (ASD) metric, a measure of how effectively a mechanism advantages truth-telling over deception. We supply a Python package to facilitate rapid and competitive evaluation of scalable oversight protocols on our benchmark, and conduct a demonstrative experiment benchmarking Debate.