# Least Square Calibration for Peer Review

arXiv: https://arxiv.org/abs/2110.12607 · HTML: https://mlsanity.com/paper/2110.12607/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 7 of 7)

- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings](https://mlsanity.com/paper/1806.05085/) (2018)
- [Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment](https://mlsanity.com/paper/2010.04041/) (2020)
- [Debiasing Evaluations That are Biased by Evaluations](https://mlsanity.com/paper/2012.00714/) (2020)
- [Calibration with confidence: A principled method for panel assessment](https://mlsanity.com/paper/1504.00340/) (2015)
- [Report-Sensitive Spot-Checking in Peer-Grading Systems](https://mlsanity.com/paper/1906.05884/) (2019)
- [Peer Grading in a Course on Algorithms and Data Structures: Machine Learning Algorithms do not Improve over Simple Baselines](https://mlsanity.com/paper/1506.00852/) (2015)

## Built on by (most cited first, 3 of 3)

- [OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models](https://mlsanity.com/paper/2505.21537/) (2025)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [The IJCNN 2025 Review Process](https://mlsanity.com/paper/2603.19244/) (2026)

## Abstract (arXiv)

Peer review systems such as conference paper review often suffer from the issue of miscalibration. Previous works on peer review calibration usually only use the ordinal information or assume simplistic reviewer scoring functions such as linear functions. In practice, applications like academic conferences often rely on manual methods, such as open discussions, to mitigate miscalibration. It remains an important question to develop algorithms that can handle different types of miscalibrations based on available prior knowledge. In this paper, we propose a flexible framework, namely least square calibration (LSC), for selecting top candidates from peer ratings. Our framework provably performs perfect calibration from noiseless linear scoring functions under mild assumptions, yet also provides competitive calibration results when the scoring function is from broader classes beyond linear functions and with arbitrary noise. On our synthetic dataset, we empirically demonstrate that our algorithm consistently outperforms the baseline which select top papers based on the highest average ratings.