# Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments

arXiv: https://arxiv.org/abs/2311.09497 · HTML: https://mlsanity.com/paper/2311.09497/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2023.

## Built on by (most cited first, 27 of 27)

- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews](https://mlsanity.com/paper/2408.10365/) (2024)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Benchmarking LLMs' Judgments with No Gold Standard](https://mlsanity.com/paper/2411.07127/) (2024)
- [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026)
- [LLMs as Meta-Reviewers' Assistants: A Case Study](https://mlsanity.com/paper/2402.15589/) (2024)
- [Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment](https://mlsanity.com/paper/2411.03417/) (2024)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers](https://mlsanity.com/paper/2511.21843/) (2025)
- [Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem](https://mlsanity.com/paper/2506.08134/) (2025)
- [The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review](https://mlsanity.com/paper/2408.13430/) (2024)
- [Identifying Aspects in Peer Reviews](https://mlsanity.com/paper/2504.06910/) (2025)
- [Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions](https://mlsanity.com/paper/2508.13214/) (2025)
- [RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation](https://mlsanity.com/paper/2601.04350/) (2026)
- [PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality](https://mlsanity.com/paper/2606.20897/) (2026)
- [Is Peer-Reviewing Worth the Effort?](https://mlsanity.com/paper/2412.14351/) (2024)
- [ReviewScore: Misinformed Peer Review Detection with Large Language Models](https://mlsanity.com/paper/2509.21679/) (2025)
- [CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers](https://mlsanity.com/paper/2605.07905/) (2026)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review](https://mlsanity.com/paper/2501.13014/) (2025)
- [Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time](https://mlsanity.com/paper/2601.15172/) (2026)
- [WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews](https://mlsanity.com/paper/2601.17377/) (2026)
- [Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance](https://mlsanity.com/paper/2602.10118/) (2026)
- [Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement](https://mlsanity.com/paper/2606.22166/) (2026)
- [Position: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System](https://mlsanity.com/paper/2608.14571/) (2026)
- [Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviews](https://mlsanity.com/paper/2609.08475/) (2026)

## Abstract (arXiv)

Is it possible to reliably evaluate the quality of peer reviews? We study this question driven by two primary motivations -- incentivizing high-quality reviewing using assessed quality of reviews and measuring changes to review quality in experiments. We conduct a large scale study at the NeurIPS 2022 conference, a top-tier conference in machine learning, in which we invited (meta)-reviewers and authors to evaluate reviews given to submitted papers. First, we conduct a RCT to examine bias due to the length of reviews. We generate elongated versions of reviews by adding substantial amounts of non-informative content. Participants in the control group evaluate the original reviews, whereas participants in the experimental group evaluate the artificially lengthened versions. We find that lengthened reviews are scored (statistically significantly) higher quality than the original reviews. In analysis of observational data we find that authors are positively biased towards reviews recommending acceptance of their own papers, even after controlling for confounders of review length, quality, and different numbers of papers per author. We also measure disagreement rates between multiple evaluations of the same review of 28%-32%, which is comparable to that of paper reviewers at NeurIPS. Further, we assess the amount of miscalibration of evaluators of reviews using a linear model of quality scores and find that it is similar to estimates of miscalibration of paper reviewers at NeurIPS. Finally, we estimate the amount of variability in subjective opinions around how to map individual criteria to overall scores of review quality and find that it is roughly the same as that in the review of papers. Our results suggest that the various problems that exist in reviews of papers -- inconsistency, bias towards irrelevant factors, miscalibration, subjectivity -- also arise in reviewing of reviews.