mlsanityData to 14 Oct 2026

Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem

arXiv 2506.08134 ↗First posted 9 Jun 2025cs.AI cs.CYcited by 9

Not on the map. Why?

Abstract (arXiv)

Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale. Exponential growth in manuscript submissions to premier ML venues such as NeurIPS, ICML, and ICLR is outpacing the finite capacity of qualified reviewers, leading to concerns about review quality, consistency, and reviewer fatigue. This position paper argues that AI-assisted peer review must become an urgent research and infrastructure priority. We advocate for a comprehensive AI-augmented ecosystem, leveraging Large Language Models (LLMs) not as replacements for human judgment, but as sophisticated collaborators for authors, reviewers, and Area Chairs (ACs). We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making. Crucially, we contend that the development of such systems hinges on access to more granular, structured, and ethically-sourced peer review process data. We outline a research agenda, including illustrative experiments, to develop and validate these AI assistants, and discuss significant technical and ethical challenges. We call upon the ML community to proactively build this AI-assisted future, ensuring the continued integrity and scalability of scientific validation, while maintaining high standards of peer review.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

25 of 44 references.

References · 25

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  3. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  4. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  5. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  6. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  7. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  8. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  9. 2024MARG: Multi-Agent Review Generation for Scientific Paperscited by 104
  10. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  11. 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
  12. 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 53
13 more
  1. 2025Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025cited by 47
  2. 2023Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experimentcited by 34
  3. 2025Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewardscited by 34
  4. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  5. 2023Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experimentscited by 27
  6. 2024Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attackcited by 27
  7. 2024Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Reviewcited by 20
  8. 2023Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPTcited by 19
  9. 2019A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniquescited by 15
  10. 2024LLM4ED: Large Language Models for Automatic Equation Discoverycited by 14
  11. 2024Automatically Learning Hybrid Digital Twins of Dynamical Systemscited by 14
  12. 2024LitLLMs, LLMs for Literature Review: Are we there yet?cited by 14
  13. 2025Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Actingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.