# Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

arXiv: https://arxiv.org/abs/2306.03262 · HTML: https://mlsanity.com/paper/2306.03262/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 15 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 2 of 2)

- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?](https://mlsanity.com/paper/2211.12966/) (2022)

## Built on by (most cited first, 30 of 34)

- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback](https://mlsanity.com/paper/2410.04025/) (2024)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback](https://mlsanity.com/paper/2508.10795/) (2025)
- [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025)
- [Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews](https://mlsanity.com/paper/2509.13400/) (2025)
- [Can AI agents conduct open-ended AI research? Early evidence from two case studies](https://mlsanity.com/paper/2607.27191/) (2026)
- [Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem](https://mlsanity.com/paper/2506.08134/) (2025)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [FARS: A Fully Automated Research System Deployed at Scale](https://mlsanity.com/paper/2606.31651/) (2026)
- [Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists](https://mlsanity.com/paper/2604.28158/) (2026)
- [On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists](https://mlsanity.com/paper/2605.20668/) (2026)
- [On the Rigour of Scientific Writing: Criteria, Analysis, and Insights](https://mlsanity.com/paper/2410.04981/) (2024)
- [Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning](https://mlsanity.com/paper/2512.02914/) (2025)
- [Learning the Bitter Lesson: Empirical Evidence from 20 Years of CVPR Proceedings](https://mlsanity.com/paper/2410.09649/) (2024)
- [How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review](https://mlsanity.com/paper/2510.02143/) (2025)
- [RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation](https://mlsanity.com/paper/2601.04350/) (2026)
- [APRES: An Agentic Paper Revision and Evaluation System](https://mlsanity.com/paper/2603.03142/) (2026)
- [PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers](https://mlsanity.com/paper/2605.26730/) (2026)
- [Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process](https://mlsanity.com/paper/2502.00874/) (2025)
- [What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review](https://mlsanity.com/paper/2604.19998/) (2026)
- [E3: Issue-Level Backtesting for Automated Research Critique](https://mlsanity.com/paper/2605.27072/) (2026)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [GUIDE: Towards Scalable Advising for Research Ideas](https://mlsanity.com/paper/2507.08870/) (2025)
- [Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism](https://mlsanity.com/paper/2601.15249/) (2026)
- [Review Arcade: On the Human Alignment and Gameability of LLM Reviews](https://mlsanity.com/paper/2605.28897/) (2026)
- [Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time](https://mlsanity.com/paper/2601.15172/) (2026)
- [Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes](https://mlsanity.com/paper/2606.15887/) (2026)
- [Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement](https://mlsanity.com/paper/2606.22166/) (2026)

## Abstract (arXiv)

We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun. Our analysis suggests that making the conference more selective would increase the arbitrariness of the process. Taken together with previous research, our results highlight the inherent difficulty of objectively measuring the quality of research, and suggest that authors should not be excessively discouraged by rejected work.