# ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews

arXiv: https://arxiv.org/abs/2503.08506 · HTML: https://mlsanity.com/paper/2503.08506/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training](https://mlsanity.com/paper/2406.16554/) (2024)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents](https://mlsanity.com/paper/2410.13185/) (2024)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [MReD: A Meta-Review Dataset for Structure-Controllable Text Generation](https://mlsanity.com/paper/2110.07474/) (2021)
- [GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study](https://mlsanity.com/paper/2307.05492/) (2023)
- [Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs](https://mlsanity.com/paper/2406.14544/) (2024)
- [VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models](https://mlsanity.com/paper/2503.07478/) (2025)
- [From Words to Worth: Newborn Article Impact Prediction with LLM](https://mlsanity.com/paper/2408.03934/) (2024)
- [MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios](https://mlsanity.com/paper/2409.16084/) (2024)
- [Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report](https://mlsanity.com/paper/2406.11403/) (2024)
- [FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion](https://mlsanity.com/paper/2410.12564/) (2024)

## Built on by (most cited first, 30 of 32)

- [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [EIAD: Explainable Industrial Anomaly Detection Via Multi-Modal Large Language Models](https://mlsanity.com/paper/2503.14162/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review](https://mlsanity.com/paper/2604.09590/) (2026)
- [DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning](https://mlsanity.com/paper/2601.07611/) (2026)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges](https://mlsanity.com/paper/2505.15068/) (2025)
- [A Multi-Task Evaluation of LLMs' Processing of Academic Text Input](https://mlsanity.com/paper/2508.11779/) (2025)
- [ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review](https://mlsanity.com/paper/2601.22638/) (2026)
- [LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers](https://mlsanity.com/paper/2605.25415/) (2026)
- [What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review](https://mlsanity.com/paper/2604.19998/) (2026)
- [CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers](https://mlsanity.com/paper/2605.07905/) (2026)
- [PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing](https://mlsanity.com/paper/2605.29815/) (2026)
- [RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review](https://mlsanity.com/paper/2608.00005/) (2026)
- [TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews](https://mlsanity.com/paper/2605.26911/) (2026)
- [PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf](https://mlsanity.com/paper/2606.08857/) (2026)
- [Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review](https://mlsanity.com/paper/2606.12716/) (2026)
- [Benchmarking Agentic Review Systems](https://mlsanity.com/paper/2606.19749/) (2026)
- [LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges](https://mlsanity.com/paper/2606.25057/) (2026)
- [Uncertainty-Aware Generation and Decision-Making Under Ambiguity](https://mlsanity.com/paper/2606.30578/) (2026)
- [ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent System](https://mlsanity.com/paper/2607.14178/) (2026)
- [ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs](https://mlsanity.com/paper/2608.12788/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [Beyond Detection: Governing GenAI in Academic Peer Review as a Sociotechnical Challenge](https://mlsanity.com/paper/2603.20214/) (2026)
- [AgentEconomist: An End-to-end Agentic System Translating Economic Intuitions into Executable Computational Experiments](https://mlsanity.com/paper/2604.27725/) (2026)
- [Judgment-Grounded Expansion for Peer Review Generation](https://mlsanity.com/paper/2606.23233/) (2026)
- [From peer review nuances to best practices](https://mlsanity.com/paper/2607.22681/) (2026)
- [ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation](https://mlsanity.com/paper/2609.09076/) (2026)

## Abstract (arXiv)

Academic paper review is a critical yet time-consuming task within the research community. With the increasing volume of academic publications, automating the review process has become a significant challenge. The primary issue lies in generating comprehensive, accurate, and reasoning-consistent review comments that align with human reviewers' judgments. In this paper, we address this challenge by proposing ReviewAgents, a framework that leverages large language models (LLMs) to generate academic paper reviews. We first introduce a novel dataset, Review-CoT, consisting of 142k review comments, designed for training LLM agents. This dataset emulates the structured reasoning process of human reviewers-summarizing the paper, referencing relevant works, identifying strengths and weaknesses, and generating a review conclusion. Building upon this, we train LLM reviewer agents capable of structured reasoning using a relevant-paper-aware training method. Furthermore, we construct ReviewAgents, a multi-role, multi-LLM agent review framework, to enhance the review comment generation process. Additionally, we propose ReviewBench, a benchmark for evaluating the review comments generated by LLMs. Our experimental results on ReviewBench demonstrate that while existing LLMs exhibit a certain degree of potential for automating the review process, there remains a gap when compared to human-generated reviews. Moreover, our ReviewAgents framework further narrows this gap, outperforming advanced LLMs in generating review comments.