mlsanityData to 14 Oct 2026

Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

arXiv 2306.03262 ↗First posted 5 Jun 2023cs.LG cs.DLcited by 34

Not on the map. Why?

Abstract (arXiv)

We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun. Our analysis suggests that making the conference more selective would increase the arbitrariness of the process. Taken together with previous research, our results highlight the inherent difficulty of objectively measuring the quality of research, and suggest that authors should not be excessively discouraged by rejected work.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 2 of its 15 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

2 of 15 references.

References · 2

  1. 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
  2. 2022How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?cited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 19.

34 papers.

Sort:

By area:

  1. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  2. 2024IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedbackcited by 42
  3. 2025Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewardscited by 34
  4. 2025How Far Are AI Scientists from Changing the World?cited by 15
  5. 2025Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedbackcited by 12
  6. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  7. 2025Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviewscited by 11
  8. 2026Can AI agents conduct open-ended AI research? Early evidence from two case studiescited by 10
  9. 2025Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystemcited by 9
  10. 2024The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Reviewcited by 8
  11. 2026FARS: A Fully Automated Research System Deployed at Scalecited by 7
  12. 2026Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientistscited by 6
22 more
  1. 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
  2. 2024On the Rigour of Scientific Writing: Criteria, Analysis, and Insightscited by 5
  3. 2025Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoningcited by 5
  4. 2024Learning the Bitter Lesson: Empirical Evidence from 20 Years of CVPR Proceedingscited by 4
  5. 2025How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Reviewcited by 4
  6. 2026RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluationcited by 3
  7. 2026APRES: An Agentic Paper Revision and Evaluation Systemcited by 3
  8. 2026PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewerscited by 3
  9. 2025Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Processcited by 2
  10. 2026What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Reviewcited by 2
  11. 2026E3: Issue-Level Backtesting for Automated Research Critiquecited by 2
  12. 2026Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Reviewcited by 2
  13. 2025GUIDE: Towards Scalable Advising for Research Ideascited by 1
  14. 2026Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanismcited by 1
  15. 2026Review Arcade: On the Human Alignment and Gameability of LLM Reviewscited by 1
  16. 2026Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Timecited by 0
  17. 2026Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomescited by 0
  18. 2026Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movementcited by 0
  19. 2026Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit Systemcited by 0
  20. 2026More Criticism Does Not Make a Better Review: EquiReview-Rcited by 0
  21. 2026Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviewscited by 0
  22. 2026The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)cited by 0