# Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

arXiv: https://arxiv.org/abs/1806.05085 · HTML: https://mlsanity.com/paper/1806.05085/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2018.

## Built on by (most cited first, 28 of 28)

- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [Direct Judgement Preference Optimization](https://mlsanity.com/paper/2409.14664/) (2024)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025)
- [Federated Best Arm Identification with Heterogeneous Clients](https://mlsanity.com/paper/2210.07780/) (2022)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise](https://mlsanity.com/paper/2201.10631/) (2022)
- [How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?](https://mlsanity.com/paper/2211.12966/) (2022)
- [Isotonic Mechanism for Exponential Family Estimation in Machine Learning Peer Review](https://mlsanity.com/paper/2304.11160/) (2023)
- [Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective](https://mlsanity.com/paper/2109.01190/) (2021)
- [Least Square Calibration for Peer Review](https://mlsanity.com/paper/2110.12607/) (2021)
- [Perceptual adjustment queries and an inverted measurement paradigm for low-rank metric learning](https://mlsanity.com/paper/2309.04626/) (2023)
- [Peer Selection with Noisy Assessments](https://mlsanity.com/paper/2107.10121/) (2021)
- [Auctions and Peer Prediction for Academic Peer Review](https://mlsanity.com/paper/2109.00923/) (2021)
- [Subset Selection Based On Multiple Rankings in the Presence of Bias: Effectiveness of Fairness Constraints for Multiwinner Voting Score Functions](https://mlsanity.com/paper/2306.09835/) (2023)
- [A Unified Statistical Learning Model for Rankings and Scores with Application to Grant Panel Review](https://mlsanity.com/paper/2201.02539/) (2022)
- [Manipulation and Peer Mechanisms: A Survey](https://mlsanity.com/paper/2210.01984/) (2022)
- [Personalized Recommendations via Active Utility-based Pairwise Sampling](https://mlsanity.com/paper/2508.14911/) (2025)
- [Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism](https://mlsanity.com/paper/2601.15249/) (2026)
- [Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?](https://mlsanity.com/paper/2209.08665/) (2022)
- [No Agreement Without Loss: Learning and Social Choice in Peer Review](https://mlsanity.com/paper/2211.02144/) (2022)
- [Eliciting Honest Information From Authors Using Sequential Review](https://mlsanity.com/paper/2311.14619/) (2023)
- [Recycling History: Efficient Recommendations from Contextual Dueling Bandits](https://mlsanity.com/paper/2508.18841/) (2025)
- [Ranking Items from Discrete Ratings: The Cost of Unknown User Thresholds](https://mlsanity.com/paper/2510.01871/) (2025)
- [Which Coauthor Should I Nominate in My 99 ICLR Submissions? A Mathematical Analysis of the ICLR 2026 Reciprocal Reviewer Nomination Policy](https://mlsanity.com/paper/2512.17950/) (2025)
- [The IJCNN 2025 Review Process](https://mlsanity.com/paper/2603.19244/) (2026)
- [One Score, Two Decisions: Selective Prediction on the Rare-Disease Tail](https://mlsanity.com/paper/2608.14683/) (2026)

## Abstract (arXiv)

Cardinal scores (numeric ratings) collected from people are well known to suffer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores. This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood. In the absence of simplifying assumptions on the miscalibration, it is widely believed by the crowdsourcing community that the only useful information in the cardinal scores is the induced ranking. In this paper, inspired by the framework of Stein's shrinkage, empirical Bayes, and the classic two-envelope problem, we contest this widespread belief. Specifically, we consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations which are only required to be consistent with the induced ranking. We design estimators which despite making no assumptions on the miscalibration, strictly and uniformly outperform all possible estimators that rely on only the ranking. Our estimators are flexible in that they can be used as a plug-in for a variety of applications, and we provide a proof-of-concept for A/B testing and ranking. Our results thus provide novel insights in the eternal debate between cardinal and ordinal data.