Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem
Not on the map. Why?
Abstract (arXiv)
Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale. Exponential growth in manuscript submissions to premier ML venues such as NeurIPS, ICML, and ICLR is outpacing the finite capacity of qualified reviewers, leading to concerns about review quality, consistency, and reviewer fatigue. This position paper argues that AI-assisted peer review must become an urgent research and infrastructure priority. We advocate for a comprehensive AI-augmented ecosystem, leveraging Large Language Models (LLMs) not as replacements for human judgment, but as sophisticated collaborators for authors, reviewers, and Area Chairs (ACs). We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making. Crucially, we contend that the development of such systems hinges on access to more granular, structured, and ethically-sourced peer review process data. We outline a research agenda, including illustrative experiments, to develop and validate these AI assistants, and discuss significant technical and ethical challenges. We call upon the ML community to proactively build this AI-assisted future, ensuring the continued integrity and scalability of scientific validation, while maintaining high standards of peer review.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 9 Jun 2025.
Builds on
25 of 44 references.
References · 25
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
- 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
- 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
- 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
- 2022A Survey on Retrieval-Augmented Text Generationcited by 158
- 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
- 2024MARG: Multi-Agent Review Generation for Scientific Paperscited by 104
- 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
- 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
- 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 53
13 more
- 2025Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025cited by 47
- 2023Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experimentcited by 34
- 2025Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewardscited by 34
- 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
- 2023Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experimentscited by 27
- 2024Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attackcited by 27
- 2024Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Reviewcited by 20
- 2023Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPTcited by 19
- 2019A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniquescited by 15
- 2024LLM4ED: Large Language Models for Automatic Equation Discoverycited by 14
- 2024Automatically Learning Hybrid Digital Twins of Dynamical Systemscited by 14
- 2024LitLLMs, LLMs for Literature Review: Are we there yet?cited by 14
- 2025Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Actingcited by 6
Built on by
Citing papers per year: 2025: 3; 2026: 6.
9 papers.
- 2025Insights from the ICLR Peer Review and Rebuttal Processcited by 11
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
- 2025Fact-Augmented Lookahead Planning for LLM Agentscited by 8
- 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
- 2025ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentationcited by 3
- 2026ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Reviewcited by 1
- 2026Review Arcade: On the Human Alignment and Gameability of LLM Reviewscited by 1
- 2026Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliancecited by 0
- 2026Toward an Engineering of Science: Rebalancing Generation and Verification in the Age of AIcited by 0
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap29 Jun 2026
- 2026Review Arcade: On the Human Alignment and Gameability of LLM Reviews27 May 2026
- 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists20 May 2026
- 2026Toward an Engineering of Science: Rebalancing Generation and Verification in the Age of AI11 May 2026
- 2026ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review4 May 2026
- 2026Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance17 Jan 2026
- 2025Insights from the ICLR Peer Review and Rebuttal Process19 Nov 2025
- 2025ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation18 Oct 2025
- 2025Fact-Augmented Lookahead Planning for LLM Agents10 Jun 2025