# Peer Selection with Noisy Assessments

arXiv: https://arxiv.org/abs/2107.10121 · HTML: https://mlsanity.com/paper/2107.10121/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings](https://mlsanity.com/paper/1806.05085/) (2018)
- [On Strategyproof Conference Peer Review](https://mlsanity.com/paper/1806.06266/) (2018)
- [Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment](https://mlsanity.com/paper/2010.04041/) (2020)
- [Strategyproof Peer Selection using Randomization, Partitioning, and Apportionment](https://mlsanity.com/paper/1604.03632/) (2016)
- [Tuned Models of Peer Assessment in MOOCs](https://mlsanity.com/paper/1307.2579/) (2013)
- [Debiasing Evaluations That are Biased by Evaluations](https://mlsanity.com/paper/2012.00714/) (2020)
- [The PeerRank Method for Peer Assessment](https://mlsanity.com/paper/1405.7192/) (2014)
- [The Conference Paper Assignment Problem: Using Order Weighted Averages to Assign Indivisible Goods](https://mlsanity.com/paper/1705.06840/) (2017)
- [PeerNomination: Relaxing Exactness for Increased Accuracy in Peer Selection](https://mlsanity.com/paper/2004.14939/) (2020)
- [Optimal Impartial Selection](https://mlsanity.com/paper/1310.8631/) (2013)
- [A Near-Optimal Mechanism for Impartial Selection](https://mlsanity.com/paper/1407.8535/) (2014)
- [Aggregating partial rankings with applications to peer grading in massive online open courses](https://mlsanity.com/paper/1411.4619/) (2014)
- [Unbiased Comparative Evaluation of Ranking Functions](https://mlsanity.com/paper/1604.07209/) (2016)

## Built on by (most cited first, 2 of 2)

- [Manipulation and Peer Mechanisms: A Survey](https://mlsanity.com/paper/2210.01984/) (2022)
- [Towards Group Learning: Distributed Weighting of Experts](https://mlsanity.com/paper/2206.02566/) (2022)

## Abstract (arXiv)

In the peer selection problem a group of agents must select a subset of themselves as winners for, e.g., peer-reviewed grants or prizes. Here, we take a Condorcet view of this aggregation problem, i.e., that there is a ground-truth ordering over the agents and we wish to select the best set of agents, subject to the noisy assessments of the peers. Given this model, some agents may be unreliable, while others might be self-interested, attempting to influence the outcome in their favour. In this paper we extend PeerNomination, the most accurate peer reviewing algorithm to date, into WeightedPeerNomination, which is able to handle noisy and inaccurate agents. To do this, we explicitly formulate assessors' reliability weights in a way that does not violate strategyproofness, and use this information to reweight their scores. We show analytically that a weighting scheme can improve the overall accuracy of the selection significantly. Finally, we implement several instances of reweighting methods and show empirically that our methods are robust in the face of noisy assessments.