mlsanityData to 14 Oct 2026

ReviewEval: An Evaluation Framework for AI-Generated Reviews

arXiv 2502.11736 ↗First posted 17 Feb 2025cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

The escalating volume of academic research, coupled with a shortage of qualified reviewers, necessitates innovative approaches to peer review. In this work, we propose: 1. ReviewEval, a comprehensive evaluation framework for AI-generated reviews that measures alignment with human assessments, verifies factual accuracy, assesses analytical depth, identifies degree of constructiveness and adherence to reviewer guidelines; and 2. ReviewAgent, an LLM-based review generation agent featuring a novel alignment mechanism to tailor feedback to target conferences and journals, along with a self-refinement loop that iteratively optimizes its intermediate outputs and an external improvement loop using ReviewEval to improve upon the final reviews. ReviewAgent improves actionable insights by 6.78% and 47.62% over existing AI baselines and expert reviews respectively. Further, it boosts analytical depth by 3.97% and 12.73%, enhances adherence to guidelines by 10.11% and 47.26% respectively. This paper establishes essential metrics for AIbased peer review and substantially enhances the reliability and impact of AI-generated reviews in academic research.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 15 of its 63 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

15 of 63 references.

References · 15

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  6. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  7. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  8. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  9. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  10. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 69
  11. 2025Large language models for automated scholarly paper review: A surveycited by 61
  12. 2023GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Studycited by 40
3 more
  1. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  2. 2022MOPRD: A multidisciplinary open peer review datasetcited by 32
  3. 2024Teaching Language Models to Self-Improve by Learning from Language Feedbackcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 18.

25 papers.

Sort:

By area:

  1. 2025ReviewRL: Towards Automated Scientific Review with RLcited by 19
  2. 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 19
  3. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 14
  4. 2025Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Paperscited by 12
  5. 2025REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learningcited by 11
  6. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  7. 2026DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Reviewcited by 7
  8. 2025When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injectioncited by 6
  9. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  10. 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
  11. 2025"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewerscited by 5
  12. 2026DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttalcited by 5
13 more
  1. 2026ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Reviewcited by 4
  2. 2026Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Reviewcited by 3
  3. 2026From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluationcited by 2
  4. 2026What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Reviewcited by 2
  5. 2026CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewerscited by 2
  6. 2026FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Researchcited by 1
  7. 2026Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AIcited by 1
  8. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  9. 2026GraphReview: Scientific Paper Evaluation via LLM-based Graph Evidence Expansioncited by 0
  10. 2026InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttalcited by 0
  11. 2026Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Reviewcited by 0
  12. 2026Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metricscited by 0
  13. 2026Can We Believe AI Peer Review?cited by 0