# The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

arXiv: https://arxiv.org/abs/2408.13430 · HTML: https://mlsanity.com/paper/2408.13430/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 24 Aug 2024.

## Builds on (most cited first, 23 of 23)

- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments](https://mlsanity.com/paper/2006.16437/) (2020)
- [Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment](https://mlsanity.com/paper/2306.03262/) (2023)
- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [Paper Matching with Local Fairness Constraints](https://mlsanity.com/paper/1905.11924/) (2019)
- [Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings](https://mlsanity.com/paper/1806.05085/) (2018)
- [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023)
- [No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"](https://mlsanity.com/paper/2209.14987/) (2022)
- [Ranking Inferences Based on the Top Choice of Multiway Comparisons](https://mlsanity.com/paper/2211.11957/) (2022)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences](https://mlsanity.com/paper/2011.15050/) (2020)
- [Strategyproof Peer Selection using Randomization, Partitioning, and Apportionment](https://mlsanity.com/paper/1604.03632/) (2016)
- [Spectral Ranking Inferences based on General Multiway Comparisons](https://mlsanity.com/paper/2308.02918/) (2023)
- [Avoiding a Tragedy of the Commons in the Peer Review Process](https://mlsanity.com/paper/1901.06246/) (2018)
- [Debiasing Evaluations That are Biased by Evaluations](https://mlsanity.com/paper/2012.00714/) (2020)
- [PeerNomination: Relaxing Exactness for Increased Accuracy in Peer Selection](https://mlsanity.com/paper/2004.14939/) (2020)
- [How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?](https://mlsanity.com/paper/2211.12966/) (2022)
- [Isotonic Mechanism for Exponential Family Estimation in Machine Learning Peer Review](https://mlsanity.com/paper/2304.11160/) (2023)
- [Some Ethical Issues in the Review Process of Machine Learning Conferences](https://mlsanity.com/paper/2106.00810/) (2021)
- [Peer Prediction for Peer Review: Designing a Marketplace for Ideas](https://mlsanity.com/paper/2303.16855/) (2023)

## Built on by (most cited first, 8 of 8)

- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review](https://mlsanity.com/paper/2510.02143/) (2025)
- [Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism](https://mlsanity.com/paper/2601.15249/) (2026)
- [Strategic Candidacy in Generative AI Arenas](https://mlsanity.com/paper/2603.26891/) (2026)
- [From Authors to Reviewers: Leveraging Rankings to Improve Peer Review](https://mlsanity.com/paper/2510.21726/) (2025)
- [Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking](https://mlsanity.com/paper/2602.06078/) (2026)
- [Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System](https://mlsanity.com/paper/2608.14571/) (2026)

## Abstract (arXiv)

We conducted an experiment during the review process of the 2023 International Conference on Machine Learning (ICML), asking authors with multiple submissions to rank their papers based on perceived quality. In total, we received 1,342 rankings, each from a different author, covering 2,592 submissions. In this paper, we present an empirical analysis of how author-provided rankings could be leveraged to improve peer review processes at machine learning conferences. We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings. Our analysis shows that these ranking-calibrated scores outperform the raw review scores in estimating the ground truth ``expected review scores'' in terms of both squared and absolute error metrics. Furthermore, we propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review, including supporting senior area chairs in overseeing area chairs' recommendations, assisting in the selection of paper awards, and guiding the recruitment of emergency reviewers.