# Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

arXiv: https://arxiv.org/abs/2601.15172 · HTML: https://mlsanity.com/paper/2601.15172/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023)
- [The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates](https://mlsanity.com/paper/2405.02150/) (2024)
- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025](https://mlsanity.com/paper/2504.09737/) (2025)
- [Argument Mining for Understanding Peer Reviews](https://mlsanity.com/paper/1903.10104/) (2019)
- [Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review](https://mlsanity.com/paper/2204.10805/) (2022)
- [Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment](https://mlsanity.com/paper/2306.03262/) (2023)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [What Can We Do to Improve Peer Review in NLP?](https://mlsanity.com/paper/2010.03863/) (2020)
- [MOPRD: A multidisciplinary open peer review dataset](https://mlsanity.com/paper/2212.04972/) (2022)
- [Does My Rebuttal Matter? Insights from a Major NLP Conference](https://mlsanity.com/paper/1903.11367/) (2019)
- [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews](https://mlsanity.com/paper/2504.11042/) (2025)
- [Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond](https://mlsanity.com/paper/2201.11443/) (2022)
- [Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision](https://mlsanity.com/paper/2406.00197/) (2024)

## Abstract (arXiv)

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.