# Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction

arXiv: https://arxiv.org/abs/2601.20299 · HTML: https://mlsanity.com/paper/2601.20299/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Builds on (most cited first, 30 of 38)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [Scalable AI Safety via Doubly-Efficient Debate](https://mlsanity.com/paper/2311.14125/) (2023)
- [High-Dimension Human Value Representation in Large Language Models](https://mlsanity.com/paper/2404.07900/) (2024)

## Built on by (most cited first, 2 of 2)

- [Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach](https://mlsanity.com/paper/2608.25267/) (2026)
- [How the Audit Rule Shapes Faithful Factor Explanations in LLMs](https://mlsanity.com/paper/2610.01514/) (2026)

## Abstract (arXiv)

The evaluation and post-training of large language models (LLMs) rely on supervision, but strong supervision for difficult tasks is often unavailable, especially when evaluating frontier models. In such cases, models are demonstrated to exploit evaluations built on such imperfect supervision, leading to deceptive results. However, underutilized in LLM research, a wealth of mechanism design research focuses on game-theoretic incentive compatibility, i.e., eliciting honest and informative answers with weak supervision. Drawing from this literature, we introduce the peer prediction method for model evaluation and post-training. It rewards honest and informative answers over deceptive and uninformative ones, using a metric based on mutual predictability and without requiring ground truth labels. We demonstrate the method's effectiveness and resistance to deception, with both theoretical guarantees and empirical validation on models with up to 405B parameters. We show that training an 8B model with peer prediction-based reward recovers most of the drop in truthfulness due to prior malicious finetuning, even when the reward is produced by a 0.135B language model with no finetuning. On the evaluation front, in contrast to LLM-as-a-Judge which requires strong and trusted judges, we discover an inverse scaling property in peer prediction, where, surprisingly, resistance to deception is strengthened as the capability gap between the experts and participants widens, enabling reliable evaluation of strong models with weak supervision. In particular, LLM-as-a-Judge become worse than random guess when facing deceptive models 5-20x the judge's size, while peer prediction thrives when such gaps are large, including in cases with over 100x size difference.