# Knowledge Divergence and the Value of Debate for Scalable Oversight

arXiv: https://arxiv.org/abs/2603.05293 · HTML: https://mlsanity.com/paper/2603.05293/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 9 of 9)

- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Great Models Think Alike and this Undermines AI Oversight](https://mlsanity.com/paper/2502.04313/) (2025)
- [Scalable AI Safety via Doubly-Efficient Debate](https://mlsanity.com/paper/2311.14125/) (2023)
- [Avoiding Obfuscation with Prover-Estimator Debate](https://mlsanity.com/paper/2506.13609/) (2025)
- [When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?](https://mlsanity.com/paper/2510.20963/) (2025)
- [Debate is efficient with your time](https://mlsanity.com/paper/2602.08630/) (2026)
- [NP-Hard Lower Bound Complexity for Semantic Self-Verification](https://mlsanity.com/paper/2501.15446/) (2025)

## Built on by (most cited first, 2 of 2)

- [The impact of multi-agent debate protocols on debate quality: a controlled case study](https://mlsanity.com/paper/2603.28813/) (2026)
- [Measuring Collapse and Correction in Homogeneous-Panel LLM Debate](https://mlsanity.com/paper/2609.35279/) (2026)

## Abstract (arXiv)

AI safety via debate and reinforcement learning from AI feedback (RLAIF) are both proposed methods for scalable oversight of advanced AI systems, yet no formal framework relates them or characterizes when debate offers an advantage. We analyze this by parameterizing debate's value through the geometry of knowledge divergence between debating models. Using principal angles between models' representation subspaces, we prove that the debate advantage admits an exact closed form. When models share identical training corpora, debate reduces to RLAIF-like where a single-agent method recovers the same optimum. When models possess divergent knowledge, debate advantage scales with a phase transition from quadratic regime (debate offers negligible benefit) to linear regime (debate is essential). We classify three regimes of knowledge divergence (shared, one-sided, and compositional) and provide existence results showing that debate can achieve outcomes inaccessible to either model alone, alongside a negative result showing that sufficiently strong adversarial incentives cause coordination failure in the compositional regime, with a sharp threshold separating effective from ineffective debate. We offer the first formal connection between debate and RLAIF, a geometric foundation for understanding when adversarial oversight protocols are justified, and connection to the problem of eliciting latent knowledge across models with complementary information.