mlsanityData to 14 Oct 2026

Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis

arXiv 2407.12857 ↗First posted 9 Jul 2024cs.CL cs.DL cs.IRcited by 33

Not on the map. Why?

Abstract (arXiv)

In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 21 of its 109 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

21 of 109 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  11. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  12. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
9 more
  1. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  2. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  3. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  4. 2014Growth rates of modern science: A bibliometric analysis based on the number of publications and cited referencescited by 64
  5. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  6. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 54
  7. 2022MOPRD: A multidisciplinary open peer review datasetcited by 32
  8. 2022Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approachcited by 23
  9. 2024An LLM-Enhanced Adversarial Editing System for Lexical Simplificationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 15.

33 papers.

Sort:

By area:

  1. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  2. 2025Large language models for automated scholarly paper review: A surveycited by 61
  3. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  4. 2025Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussionscited by 23
  5. 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 19
  6. 2025OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientistscited by 18
  7. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12
  8. 2025Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Paperscited by 12
  9. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  10. 2025Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystemcited by 9
  11. 2026OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessmentcited by 8
  12. 2025Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Modelscited by 6
21 more
  1. 2025Scaling Laws in Scientific Discovery with AI and Robot Scientistscited by 5
  2. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  3. 2024Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Modelscited by 4
  4. 2025NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimationcited by 4
  5. 2026GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responsescited by 4
  6. 2025ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentationcited by 3
  7. 2025A Toolbox for Improving Evolutionary Prompt Searchcited by 3
  8. 2026Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviewscited by 3
  9. 2026When AI reviews science: Can we trust the referee?cited by 2
  10. 2025A Hierarchical Framework for Measuring Scientific Paper Innovation via Large Language Modelscited by 1
  11. 2026FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Researchcited by 1
  12. 2026Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AIcited by 1
  13. 2026Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Reviewcited by 1
  14. 2026LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challengescited by 1
  15. 2026CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Reviewcited by 1
  16. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  17. 2025AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinementcited by 0
  18. 2026From peer review nuances to best practicescited by 0
  19. 2026More Criticism Does Not Make a Better Review: EquiReview-Rcited by 0
  20. 2026How Much Were You Told? Measuring External Information in Peer Reviewscited by 0
  21. 2026Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewerscited by 0