# AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot

arXiv: https://arxiv.org/abs/2604.13940 · HTML: https://mlsanity.com/paper/2604.13940/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 29 of 29)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates](https://mlsanity.com/paper/2405.02150/) (2024)
- [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024)
- [CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation](https://mlsanity.com/paper/2503.22708/) (2025)
- [Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis](https://mlsanity.com/paper/2407.12857/) (2024)
- [ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews](https://mlsanity.com/paper/2503.08506/) (2025)
- [TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review](https://mlsanity.com/paper/2506.07642/) (2025)
- [Towards Execution-Grounded Automated AI Research](https://mlsanity.com/paper/2601.14525/) (2026)
- [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023)
- [ReviewEval: An Evaluation Framework for AI-Generated Reviews](https://mlsanity.com/paper/2502.11736/) (2025)
- [AAAR-1.0: Assessing AI's Potential to Assist Research](https://mlsanity.com/paper/2410.22394/) (2024)
- [Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews](https://mlsanity.com/paper/2502.17086/) (2025)
- [ResearchGym: Evaluating Language Model Agents on Real-World AI Research](https://mlsanity.com/paper/2602.15112/) (2026)
- [Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers](https://mlsanity.com/paper/2507.02694/) (2025)
- [LLMs as Meta-Reviewers' Assistants: A Case Study](https://mlsanity.com/paper/2402.15589/) (2024)
- [Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment](https://mlsanity.com/paper/2411.03417/) (2024)
- [ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review](https://mlsanity.com/paper/2510.08867/) (2025)
- [A Large Language Model Approach to Educational Survey Feedback Analysis](https://mlsanity.com/paper/2309.17447/) (2023)
- [FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers](https://mlsanity.com/paper/2511.21843/) (2025)
- [Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluation](https://mlsanity.com/paper/2505.23824/) (2025)
- [From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews](https://mlsanity.com/paper/2312.01202/) (2023)

## Built on by (most cited first, 19 of 19)

- [Stop Automating Peer Review Without Rigorous Evaluation](https://mlsanity.com/paper/2605.03202/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists](https://mlsanity.com/paper/2605.20668/) (2026)
- [From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent](https://mlsanity.com/paper/2606.13349/) (2026)
- [Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community](https://mlsanity.com/paper/2606.10159/) (2026)
- [Towards Automating Scientific Review with Google's Paper Assistant Tool](https://mlsanity.com/paper/2606.28277/) (2026)
- [Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud](https://mlsanity.com/paper/2607.10712/) (2026)
- [Review Arcade: On the Human Alignment and Gameability of LLM Reviews](https://mlsanity.com/paper/2605.28897/) (2026)
- [Benchmarking Agentic Review Systems](https://mlsanity.com/paper/2606.19749/) (2026)
- [Do Methods Support the Claims? Intra-Paper Verification for Peer Review](https://mlsanity.com/paper/2607.26066/) (2026)
- [AI-assisted pre-review of open-source software submissions: an experience report from BOSC 2026](https://mlsanity.com/paper/2607.27228/) (2026)
- [AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality](https://mlsanity.com/paper/2608.03581/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes](https://mlsanity.com/paper/2606.15887/) (2026)
- [Local AI pre-screening for human triple-blind peer review in health sciences](https://mlsanity.com/paper/2608.14625/) (2026)
- [Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review](https://mlsanity.com/paper/2609.05947/) (2026)
- [Beyond Natural Language: An Agent-Native Language for Autonomous Science](https://mlsanity.com/paper/2609.25421/) (2026)
- [Homogenization in Multi-Agent Systems](https://mlsanity.com/paper/2610.09824/) (2026)
- [How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting](https://mlsanity.com/paper/2610.18435/) (2026)

## Abstract (arXiv)

Scientific peer review faces mounting strain as submission volumes surge, making it increasingly difficult to sustain review quality, consistency, and timeliness. Recent advances in AI have led the community to consider its use in peer review, yet a key unresolved question is whether AI can generate technically sound reviews at real-world conference scale. Here we report the first large-scale field deployment of AI-assisted peer review: every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system. The system combined frontier models, tool use, and safeguards in a multi-stage process to generate reviews for all 22,977 full-review papers in less than a day. A large-scale survey of AAAI-26 authors and program committee members showed that participants not only found AI reviews useful, but actually preferred them to human reviews on key dimensions such as technical accuracy and research suggestions. We also introduce a novel benchmark and find that our system substantially outperforms a simple LLM-generated review baseline at detecting a variety of scientific weaknesses. Together, these results show that state-of-the-art AI methods can already make meaningful contributions to scientific peer review at conference scale, opening a path toward the next generation of synergistic human-AI teaming for evaluating research.