# Does double-blind peer-review reduce bias? Evidence from a top computer science conference

arXiv: https://arxiv.org/abs/2101.02701 · HTML: https://mlsanity.com/paper/2101.02701/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2021.

## Built on by (most cited first, 8 of 8)

- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review](https://mlsanity.com/paper/2309.05457/) (2023)
- [ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation](https://mlsanity.com/paper/2510.16549/) (2025)
- [Disparities in Peer Review Tone and the Role of Reviewer Anonymity](https://mlsanity.com/paper/2507.14741/) (2025)
- [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System](https://mlsanity.com/paper/2601.08829/) (2026)
- [Who Pays for Open Review? Visible Author Reputation and Its Effect on Ratings](https://mlsanity.com/paper/2609.11983/) (2026)

## Abstract (arXiv)

Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the reviewers, has been proposed as a way to reduce reviewer bias. Although intuitive, evidence for the effectiveness of double-blind peer review in reducing bias is limited and mixed. Here, we examine the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations (ICLR), a top computer science conference that changed its reviewing policy from single-blind peer review to double-blind peer review in 2018. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance decisions significantly. Nevertheless, we show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases. Specifically, papers rejected in the single-blind format are cited more than those rejected under the double-blind format, suggesting that double-blind review better identifies poorer quality papers. Interestingly, an apparently unrelated change - the change of rating scale from 10 to 4 points - likely reduced prestige bias significantly, to an extent that affected papers' acceptance. These results provide some support for the effectiveness of double-blind review in reducing prestige bias, while opening new research directions on the impact of peer review formats.