# How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review

arXiv: https://arxiv.org/abs/2510.02143 · HTML: https://mlsanity.com/paper/2510.02143/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment](https://mlsanity.com/paper/2306.03262/) (2023)
- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences](https://mlsanity.com/paper/2011.15050/) (2020)
- [Strategyproof Peer Selection using Randomization, Partitioning, and Apportionment](https://mlsanity.com/paper/1604.03632/) (2016)
- [Avoiding a Tragedy of the Commons in the Peer Review Process](https://mlsanity.com/paper/1901.06246/) (2018)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [PeerNomination: Relaxing Exactness for Increased Accuracy in Peer Selection](https://mlsanity.com/paper/2004.14939/) (2020)
- [How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?](https://mlsanity.com/paper/2211.12966/) (2022)
- [Isotonic Mechanism for Exponential Family Estimation in Machine Learning Peer Review](https://mlsanity.com/paper/2304.11160/) (2023)

## Built on by (most cited first, 4 of 4)

- [Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism](https://mlsanity.com/paper/2601.15249/) (2026)
- [Does Multi-Agent Debate Improve AI Feedback on Research Papers?](https://mlsanity.com/paper/2607.14713/) (2026)
- [Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2605.25172/) (2026)
- [Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent Systems](https://mlsanity.com/paper/2608.09964/) (2026)

## Abstract (arXiv)

Peer review in academic research aims not only to ensure factual correctness but also to identify work of high scientific potential that can shape future research directions. This task is especially critical in fast-moving fields such as artificial intelligence (AI), yet it has become increasingly difficult given the rapid growth of submissions. In this paper, we investigate an underexplored measure for identifying high-impact research: authors' own rankings of their multiple submissions to the same AI conference. Grounded in game-theoretic reasoning, we hypothesize that self-rankings are informative because authors possess unique understanding of their work's conceptual depth and long-term promise. To test this hypothesis, we conducted a large-scale experiment at a leading AI conference, where 1,342 researchers self-ranked their 2,592 submissions by perceived quality. Tracking outcomes over more than a year, we found that papers ranked highest by their authors received twice as many citations as their lowest-ranked counterparts; self-rankings were especially effective at identifying highly cited papers (those with over 150 citations). Moreover, we showed that self-rankings outperformed peer review scores in predicting future citation counts. Our results remained robust after accounting for confounders such as preprint posting time and self-citations. Together, these findings demonstrate that authors' self-rankings provide a reliable and valuable complement to peer review for identifying and elevating high-impact research in AI.