# No Agreement Without Loss: Learning and Social Choice in Peer Review

arXiv: https://arxiv.org/abs/2211.02144 · HTML: https://mlsanity.com/paper/2211.02144/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 9 of 9)

- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings](https://mlsanity.com/paper/1806.05085/) (2018)
- [On Strategyproof Conference Peer Review](https://mlsanity.com/paper/1806.06266/) (2018)
- [Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment](https://mlsanity.com/paper/2010.04041/) (2020)
- [Loss Functions, Axioms, and Peer Review](https://mlsanity.com/paper/1808.09057/) (2018)
- [Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment](https://mlsanity.com/paper/1507.06411/) (2015)
- [Partial Strategyproofness: Relaxing Strategyproofness for the Random Assignment Problem](https://mlsanity.com/paper/1401.3675/) (2014)
- [Differentially Private Condorcet Voting](https://mlsanity.com/paper/2206.13081/) (2022)
- [Do peers share the same criteria for assessing grant applications?](https://mlsanity.com/paper/2106.07386/) (2021)

## Abstract (arXiv)

In peer review systems, reviewers are often asked to evaluate various features of submissions, such as technical quality or novelty. A score is given to each of the predefined features and based on these the reviewer has to provide an overall quantitative recommendation. It may be assumed that each reviewer has her own mapping from the set of features to a recommendation, and that different reviewers have different mappings in mind. This introduces an element of arbitrariness known as commensuration bias. In this paper we discuss a framework, introduced by Noothigattu, Shah and Procaccia, and then applied by the organizers of the AAAI 2022 conference. Noothigattu, Shah and Procaccia proposed to aggregate reviewer's mapping by minimizing certain loss functions, and studied axiomatic properties of this approach, in the sense of social choice theory. We challenge several of the results and assumptions used in their work and report a number of negative results. On the one hand, we study a trade-off between some of the axioms proposed and the ability of the method to properly capture agreements of the majority of reviewers. On the other hand, we show that dropping a certain unrealistic assumption has dramatic effects, including causing the method to be discontinuous.