Can We Believe AI Peer Review?
Not on the map. Why?
Abstract (arXiv)
As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 11 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
5 of 11 references.
References · 5
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
- 2025ReviewEval: An Evaluation Framework for AI-Generated Reviewscited by 25
- 2026LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewerscited by 4
- 2026How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Reviewcited by 3
Built on by
Not cited yet.