Data to 7 Oct 2026

Prover-Verifier Games improve legibility of LLM outputs

arXiv 2407.13692 ↗First posted 18 Jul 2024cs.CLcited by 79

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One way to increase confidence in the outputs of Large Language Models (LLMs) is to support them with reasoning that is clear and easy to check -- a property we call legibility. We study legibility in the context of solving grade-school math problems and show that optimizing chain-of-thought solutions only for answer correctness can make them less legible. To mitigate the loss in legibility, we propose a training algorithm inspired by Prover-Verifier Game from Anil et al. (2021). Our algorithm iteratively trains small verifiers to predict solution correctness, "helpful" provers to produce correct solutions that the verifier accepts, and "sneaky" provers to produce incorrect solutions that fool the verifier. We find that the helpful prover's accuracy and the verifier's robustness to adversarial attacks increase over the course of training. Furthermore, we show that legibility training transfers to time-constrained humans tasked with verifying solution correctness. Over course of LLM training human accuracy increases when checking the helpful prover's solutions, and decreases when checking the sneaky prover's solutions. Hence, training for checkability by small verifiers is a plausible technique for increasing output legibility. Our results suggest legibility training against small verifiers as a practical avenue for increasing legibility of large LLMs to humans, and thus could help with alignment of superhuman models.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

48 of 64 references.

References · 48

2023GPT-4 Technical Reportcited by 18,132
2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
2013Intriguing properties of neural networkscited by 5,114
2017Deep reinforcement learning from human preferencescited by 3,846
2023Let's Verify Step by Stepcited by 3,598
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056

36 more

2023Progress measures for grokking via mechanistic interpretabilitycited by 822
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2017Evasion Attacks against Machine Learning at Test Timecited by 686
2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
2019On Evaluating Adversarial Robustnesscited by 442
2018Scalable agent alignment via reward modeling: a research directioncited by 438
2018AI safety via debatecited by 384
2022Self-critiquing models for assisting human evaluatorscited by 372
2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
2023Explainability for Large Language Models: A Surveycited by 331
2023Are aligned neural networks adversarially aligned?cited by 305
2021Recursively Summarizing Books with Human Feedbackcited by 287
2023Faithful Chain-of-Thought Reasoningcited by 285
2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
2018Supervising strong learners by amplifying weak expertscited by 139
2024LLM Critics Help Catch LLM Bugscited by 138
2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
2018Adversarial Examples that Fool both Computer Vision and Time-Limited Humanscited by 73
2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
2022Adversarial Training for High-Stakes Reliabilitycited by 60
2024Query-Based Adversarial Prompt Generationcited by 55
2023Debate Helps Supervise Unreliable Expertscited by 39
2021Learning to Give Checkable Answers with Prover-Verifier Gamescited by 29
2022Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questionscited by 16
2024Learning Task Decomposition to Assist Humans in Competitive Programmingcited by 14
2021Evaluating Code Readability and Legibility: An Examination of Human-centric Studiescited by 8
2022Interpretability Guarantees with Merlin-Arthur Classifierscited by 8
2022PAC Verification of Statistical Algorithmscited by 8
2024Models That Prove Their Own Correctnesscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 34; 2026: 36.

79 papers.

Sort:

By area:

2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
2025Why Do Multi-Agent LLM Systems Fail?cited by 559
2025Reasoning Models Don't Always Say What They Thinkcited by 388
2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
2025Evolving Deeper LLM Thinkingcited by 58
2024The Limits of Inference Scaling Through Resamplingcited by 51

38 more

2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
2025Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lenscited by 32
2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 27
2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 21
2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
2025Adversarial Training of Reward Modelscited by 17
2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signalscited by 17
2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
2025Conformal Language Model Reasoning with Coherent Factualitycited by 14
2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
2025Avoiding Obfuscation with Prover-Estimator Debatecited by 11
2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9
2025Reasoning Models Sometimes Output Illegible Chains of Thoughtcited by 9
2026Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loopscited by 8
2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Gamecited by 7
2026Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoningcited by 7
2024Retrospective Learning from Interactionscited by 6
2025Diverse Inference and Verification for Advanced Reasoningcited by 6
2025Generative Data Refinement: Just Ask for Better Datacited by 6
2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
2025Iterative Label Refinement Matters More than Preference Optimization under Weak Supervisioncited by 5
2026Measuring Weak-to-Strong Legibility of Reasoning Modelscited by 5
2026Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversightcited by 5
2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modelingcited by 4
2025Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solvingcited by 4
2024Overcoming classic challenges for artificial neural networks by providing incentives and practicecited by 3
2025FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Researchcited by 3
2026Basic Legibility Protocols Improve Trusted Monitoringcited by 3
2026Pseudo-Formalization for Automatic Proof Verificationcited by 3
2024Truth or Deceit? A Bayesian Decoding Game Enhances Consistency and Reliabilitycited by 2
2025Super Co-alignment of Human and AI for Sustainable Symbiotic Societycited by 2
2025Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulationscited by 2
2025Budget-aware Test-time Scaling via Discriminative Verificationcited by 2
2026ALIGN: Aligned Delegation with Performance Guarantees for Multi-Agent LLM Reasoningcited by 2
2026iGRPO: Self-Feedback-Driven LLM Reasoningcited by 2
2026Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Executioncited by 2
2026Agentic Systems as Boosting Weak Reasoning Modelscited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way