# Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem arXiv: https://arxiv.org/abs/2506.08134 ยท HTML: https://mlsanity.com/paper/2506.08134/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 44 references that are in the map. - First posted on arXiv 9 Jun 2025. ## Builds on (most cited first, 25 of 25) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024) - [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020) - [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025) - [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023) - [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022) - [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023) - [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024) - [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024) - [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021) - [Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review](https://mlsanity.com/paper/2412.01708/) (2024) - [Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025](https://mlsanity.com/paper/2504.09737/) (2025) - [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025) - [Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment](https://mlsanity.com/paper/2306.03262/) (2023) - [Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis](https://mlsanity.com/paper/2407.12857/) (2024) - [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023) - [Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack](https://mlsanity.com/paper/2404.01907/) (2024) - [Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review](https://mlsanity.com/paper/2410.03019/) (2024) - [Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT](https://mlsanity.com/paper/2308.05341/) (2023) - [A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques](https://mlsanity.com/paper/1907.10863/) (2019) - [LitLLMs, LLMs for Literature Review: Are we there yet?](https://mlsanity.com/paper/2412.15249/) (2024) - [LLM4ED: Large Language Models for Automatic Equation Discovery](https://mlsanity.com/paper/2405.07761/) (2024) - [Automatically Learning Hybrid Digital Twins of Dynamical Systems](https://mlsanity.com/paper/2410.23691/) (2024) - [Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Acting](https://mlsanity.com/paper/2505.24710/) (2025) ## Built on by (most cited first, 10 of 10) - [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025) - [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026) - [Fact-Augmented Lookahead Planning for LLM Agents](https://mlsanity.com/paper/2506.09171/) (2025) - [On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists](https://mlsanity.com/paper/2605.20668/) (2026) - [ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation](https://mlsanity.com/paper/2510.16549/) (2025) - [ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review](https://mlsanity.com/paper/2605.02651/) (2026) - [Review Arcade: On the Human Alignment and Gameability of LLM Reviews](https://mlsanity.com/paper/2605.28897/) (2026) - [Position on LLM-Assisted Peer Review: Addressing Reviewer Gap through Mentoring and Feedback](https://mlsanity.com/paper/2601.09182/) (2026) - [Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance](https://mlsanity.com/paper/2602.10118/) (2026) - [Toward an Engineering of Science: Rebalancing Generation and Verification in the Age of AI](https://mlsanity.com/paper/2605.10425/) (2026) ## Abstract (arXiv) Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale. Exponential growth in manuscript submissions to premier ML venues such as NeurIPS, ICML, and ICLR is outpacing the finite capacity of qualified reviewers, leading to concerns about review quality, consistency, and reviewer fatigue. This position paper argues that AI-assisted peer review must become an urgent research and infrastructure priority. We advocate for a comprehensive AI-augmented ecosystem, leveraging Large Language Models (LLMs) not as replacements for human judgment, but as sophisticated collaborators for authors, reviewers, and Area Chairs (ACs). We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making. Crucially, we contend that the development of such systems hinges on access to more granular, structured, and ethically-sourced peer review process data. We outline a research agenda, including illustrative experiments, to develop and validate these AI assistants, and discuss significant technical and ethical challenges. We call upon the ML community to proactively build this AI-assisted future, ensuring the continued integrity and scalability of scientific validation, while maintaining high standards of peer review.