# Stochastically Dominant Peer Prediction

arXiv: https://arxiv.org/abs/2506.02259 · HTML: https://mlsanity.com/paper/2506.02259/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Crowdsourced Judgement Elicitation with Endogenous Proficiency](https://mlsanity.com/paper/1303.0799/) (2013)
- [Informed Truthfulness in Multi-Task Peer Prediction](https://mlsanity.com/paper/1603.03151/) (2016)
- [Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks](https://mlsanity.com/paper/1911.00272/) (2019)
- [Tuned Models of Peer Assessment in MOOCs](https://mlsanity.com/paper/1307.2579/) (2013)
- [Learning and Strongly Truthful Multi-Task Peer Prediction: A Variational Approach](https://mlsanity.com/paper/2009.14730/) (2020)
- [Measurement Integrity in Peer Prediction: A Peer Assessment Case Study](https://mlsanity.com/paper/2108.05521/) (2021)
- [Peer Truth Serum: Incentives for Crowdsourcing Measurements and Opinions](https://mlsanity.com/paper/1704.05269/) (2017)
- [Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms](https://mlsanity.com/paper/2402.13567/) (2024)

## Built on by (most cited first, 1 of 1)

- [Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics](https://mlsanity.com/paper/2608.01423/) (2026)

## Abstract (arXiv)

Eliciting reliable human feedback is essential for many machine learning tasks, such as learning from noisy labels and aligning AI systems with human preferences. Peer prediction mechanisms incentivize truthful reporting without ground truth verification by scoring agents based on correlations with peers. Traditional mechanisms, which ensure that truth-telling maximizes the expected scores in equilibrium, can elicit honest information while assuming agents' utilities are linear functions of their scores. However, in practice, non-linear payment rules are usually preferred, or agents' utilities are inherently non-linear. We propose stochastically dominant truthfulness (SD-truthfulness) as a stronger guarantee: the score distribution of truth-telling stochastically dominates all other strategies, incentivizing truthful reporting for a wide range of monotone utility functions. Our first observation is that no existing peer prediction mechanism naturally satisfies this criterion without strong assumptions. A simple solution -- rounding scores into binary lotteries -- can enforce SD-truthfulness, but often degrades sensitivity, a key property related to fairness and statistical efficiency. We demonstrate how a more careful application of rounding can better preserve sensitivity. Furthermore, we introduce a new enforced agreement (EA) mechanism that is theoretically guaranteed to be SD-truthful in binary-signal settings under mild assumptions, and empirically achieves the highest sensitivity among all known SD-truthful mechanisms.