# An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process

arXiv: https://arxiv.org/abs/2010.05137 · HTML: https://mlsanity.com/paper/2010.05137/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [Automatic Analysis of Substantiation in Scientific Peer Reviews](https://mlsanity.com/paper/2311.11967/) (2023)
- [SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments](https://mlsanity.com/paper/2505.23713/) (2025)
- [A Siren Song of Open Source Reproducibility](https://mlsanity.com/paper/2204.04372/) (2022)
- [Position: The Current AI Conference Model is Unsustainable! Diagnosing the Crisis of Centralized AI Conference](https://mlsanity.com/paper/2508.04586/) (2025)
- [ReAct: A Review Comment Dataset for Actionability (and more)](https://mlsanity.com/paper/2210.00443/) (2022)
- [Some Ethical Issues in the Review Process of Machine Learning Conferences](https://mlsanity.com/paper/2106.00810/) (2021)
- [Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2505.15031/) (2025)
- [Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review](https://mlsanity.com/paper/2309.05457/) (2023)
- [The Ramon Llull's Thinking Machine for Automated Ideation](https://mlsanity.com/paper/2508.19200/) (2025)
- [Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process](https://mlsanity.com/paper/2502.00874/) (2025)
- [Reproducibility: The New Frontier in AI Governance](https://mlsanity.com/paper/2510.11595/) (2025)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [Artificial intelligence technologies to support research assessment: A review](https://mlsanity.com/paper/2212.06574/) (2022)
- [EchoReview: Learning Peer Review from the Echoes of Scientific Citations](https://mlsanity.com/paper/2602.00733/) (2026)
- [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026)
- [WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews](https://mlsanity.com/paper/2601.17377/) (2026)
- [Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews](https://mlsanity.com/paper/2604.07119/) (2026)
- [Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement](https://mlsanity.com/paper/2606.22166/) (2026)
- [Catalyst Papers in Artificial Intelligence Research: A Landscape on ICLR from 2017 to 2025](https://mlsanity.com/paper/2607.05401/) (2026)
- [Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees](https://mlsanity.com/paper/2608.02455/) (2026)
- [Who Pays for Open Review? Visible Author Reputation and Its Effect on Ratings](https://mlsanity.com/paper/2609.11983/) (2026)
- [The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)](https://mlsanity.com/paper/2610.06591/) (2026)

## Abstract (arXiv)

Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts. We conclude our work with recommendations for future conference organizers.