mlsanityData to 14 Oct 2026

Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025

arXiv 2504.09737 ↗First posted 13 Apr 2025cs.AI cs.CL cs.HCcited by 47

Not on the map. Why?

Abstract (arXiv)

Peer review at AI conferences is stressed by rapidly rising submission volumes, leading to deteriorating review quality and increased author dissatisfaction. To address these issues, we developed Review Feedback Agent, a system leveraging multiple large language models (LLMs) to improve review clarity and actionability by providing automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers. Implemented at ICLR 2025 as a large randomized control study, our system provided optional feedback to more than 20,000 randomly selected reviews. To ensure high-quality feedback for reviewers at this scale, we also developed a suite of automated reliability tests powered by LLMs that acted as guardrails to ensure feedback quality, with feedback only being sent to reviewers if it passed all the tests. The results show that 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers. This suggests that many reviewers found the AI-generated feedback sufficiently helpful to merit updating their reviews. Incorporating AI feedback led to significantly longer reviews (an average increase of 80 words among those who updated after receiving feedback) and more informative reviews, as evaluated by blinded researchers. Moreover, reviewers who were selected to receive AI feedback were also more engaged during paper rebuttals, as seen in longer author-reviewer discussions. This work demonstrates that carefully designed LLM-generated review feedback can enhance peer review quality by making reviews more specific and actionable while increasing engagement between reviewers and authors. The Review Feedback Agent is publicly available at https://github.com/zou-group/review_feedback_agent.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 8 of its 38 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

8 of 38 references.

References · 8

  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  3. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  4. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  5. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  6. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 118
  7. 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
  8. 2024Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experimentcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 30.

47 papers.

Sort:

By area:

  1. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  2. 2025When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Reviewcited by 19
  3. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  4. 2025Prompt Injection Attacks on LLM Generated Reviews of Scientific Publicationscited by 13
  5. 2025Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Paperscited by 12
  6. 2026SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?cited by 12
  7. 2025Insights from the ICLR Peer Review and Rebuttal Processcited by 11
  8. 2026AI Can Learn Scientific Tastecited by 11
  9. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  10. 2025In-Context Watermarks for Large Language Modelscited by 10
  11. 2025Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystemcited by 9
  12. 2025LLM-REVal: Can We Trust LLM Reviewers Yet?cited by 9
35 more
  1. 2026Paraphrasing Adversarial Attack on LLM-as-a-Reviewercited by 8
  2. 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
  3. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  4. 2025"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewerscited by 5
  5. 2025To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysiscited by 5
  6. 2025Gen-Review: A Large-scale Dataset of AI-Generated (and Human-written) Peer Reviewscited by 4
  7. 2026LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewerscited by 4
  8. 2026Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economycited by 4
  9. 2026Reflexis: Supporting Reflexivity and Rigor in Collaborative Qualitative Analysis through Design for Deliberationcited by 3
  10. 2026PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Reviewcited by 3
  11. 2026APRES: An Agentic Paper Revision and Evaluation Systemcited by 3
  12. 2026ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agentscited by 3
  13. 2025Breaking the Prompt Wall (I): A Real-World Case Study of Attacking ChatGPT via Lightweight Prompt Injectioncited by 2
  14. 2026From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluationcited by 2
  15. 2026CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewerscited by 2
  16. 2026Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraudcited by 2
  17. 2025Language Models Should be Used to Surface the Unwritten Code of Science and Societycited by 1
  18. 2026What happens when reviewers receive AI feedback in their reviews?cited by 1
  19. 2026IntelliAsk: Learning to Ask High-Quality Research Questions via RLVRcited by 1
  20. 2026Large Language Models in the Abuse Detection Pipelinecited by 1
  21. 2026ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Reviewcited by 1
  22. 2026To Trust or Not to Trust: Authors' Response to AI-based Reviewscited by 1
  23. 2026PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleafcited by 1
  24. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  25. 2025AI-Assisted Authoring for Transparent, Data-Driven Documentscited by 0
  26. 2026Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Timecited by 0
  27. 2026Beyond Detection: Governing GenAI in Academic Peer Review as a Sociotechnical Challengecited by 0
  28. 2026GraphReview: Scientific Paper Evaluation via LLM-based Graph Evidence Expansioncited by 0
  29. 2026HexEval: An Evidence-Driven Hexagonal Framework for Multidimensional Scholar Assessmentcited by 0
  30. 2026Local AI pre-screening for human triple-blind peer review in health sciencescited by 0
  31. 2026HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviewscited by 0
  32. 2026ReGround: Grounding Reviewer Comments in Multimodal Evidencecited by 0
  33. 2026PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progresscited by 0
  34. 2026How Robust Is Multimodal Claim Verification to LLM Rewriting?cited by 0
  35. 2026How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
  2. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation
  3. 2026PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
  4. 2025Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study
  5. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews
  6. 2026Benchmarking Agentic Review Systems
  7. 2026Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System
  8. 2025Dean of LLM Tutors: A Framework for Automated Quality Review of AI-generated Feedback