# Paper Copilot: Tracking the Evolution of Peer Review in AI Conferences

arXiv: https://arxiv.org/abs/2510.13201 · HTML: https://mlsanity.com/paper/2510.13201/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [MOPRD: A multidisciplinary open peer review dataset](https://mlsanity.com/paper/2212.04972/) (2022)
- [OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models](https://mlsanity.com/paper/2505.21537/) (2025)
- [The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics](https://mlsanity.com/paper/2312.04576/) (2023)
- [The Research Space: using the career paths of scholars to predict the evolution of the research output of individuals, institutions, and nations](https://mlsanity.com/paper/1602.08409/) (2016)
- [Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process](https://mlsanity.com/paper/2502.00874/) (2025)

## Built on by (most cited first, 10 of 10)

- [Stop Automating Peer Review Without Rigorous Evaluation](https://mlsanity.com/paper/2605.03202/) (2026)
- [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025)
- [Position: Academic Conferences are Potentially Facing Denominator Gaming Caused by Fully Automated Scientific Agents](https://mlsanity.com/paper/2605.09915/) (2026)
- [No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions](https://mlsanity.com/paper/2606.13044/) (2026)
- [ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues](https://mlsanity.com/paper/2606.18237/) (2026)
- [Towards Automating Scientific Review with Google's Paper Assistant Tool](https://mlsanity.com/paper/2606.28277/) (2026)
- [Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review](https://mlsanity.com/paper/2607.27209/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement](https://mlsanity.com/paper/2606.22166/) (2026)
- [Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems](https://mlsanity.com/paper/2610.01257/) (2026)

## Abstract (arXiv)

The rapid growth of AI conferences is straining an already fragile peer-review system, leading to heavy reviewer workloads, expertise mismatches, inconsistent evaluation standards, superficial or templated reviews, and limited accountability under compressed timelines. In response, conference organizers have introduced new policies and interventions to preserve review standards. Yet these ad-hoc changes often create further concerns and confusion about the review process, leaving how papers are ultimately accepted - and how practices evolve across years - largely opaque. We present Paper Copilot, a system that creates durable digital archives of peer reviews across a wide range of computer-science venues, an open dataset that enables researchers to study peer review at scale, and a large-scale empirical analysis of ICLR reviews spanning multiple years. By releasing both the infrastructure and the dataset, Paper Copilot supports reproducible research on the evolution of peer review. We hope these resources help the community track changes, diagnose failure modes, and inform evidence-based improvements toward a more robust, transparent, and reliable peer-review system.