# Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review

arXiv: https://arxiv.org/abs/2501.13014 · HTML: https://mlsanity.com/paper/2501.13014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 9 of 9)

- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Scientific Paper Recommendation: A Survey](https://mlsanity.com/paper/2008.13538/) (2020)
- [On Testing for Biases in Peer Review](https://mlsanity.com/paper/1912.13188/) (2019)
- [Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review](https://mlsanity.com/paper/2203.17239/) (2022)
- [Science Concierge: A fast content-based recommendation system for scientific publications](https://mlsanity.com/paper/1604.01070/) (2016)
- [Towards dynamic interaction-based model](https://mlsanity.com/paper/1801.03904/) (2018)
- [Reputation Gaming in Stack Overflow](https://mlsanity.com/paper/2111.07101/) (2021)

## Abstract (arXiv)

Traditional closed peer review systems, which have played a central role in scientific publishing, are often slow, costly, non-transparent, stochastic, and possibly subject to biases - factors that can impede scientific progress and undermine public trust. Here, we propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process. First, using data from two major scientific conferences (CCN2023 and ICLR2023), we highlight how high variability of review scores and low correlation across reviewers presents a challenge for collective review. We quantify reviewer agreement with community consensus scores and use this as a reviewer quality estimator, showing that surprisingly, reviewer quality scores are not correlated with authorship quality. Instead, we reveal an inverted U-shape relationship, where authors with intermediate paper scores are the best reviewers. We assess empirical Bayesian methods to estimate paper quality based on different assessments of individual reviewer reliability. We show how under a one-shot review-then-score scenario, both in our models and on real peer review data, a Bayesian measure significantly improves paper quality assessments relative to simple averaging. We then consider an ongoing model of publishing, reviewing, and scoring, with reviewers scoring not only papers but also other reviewers. We show that user-generated reviewer ratings can yield robust and high-quality paper scoring even when unreliable (but unbiased) reviewers dominate. Finally, we outline incentive structures to recognize high-quality reviewers and encourage broader reviewing coverage of submitted papers. These findings suggest that a self-selecting open peer review process is potentially scalable, reliable, and equitable with the possibility of enhancing the speed, fairness, and transparency of the peer review process.