# Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis

arXiv: https://arxiv.org/abs/2407.12857 · HTML: https://mlsanity.com/paper/2407.12857/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 109 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Focused Transformer: Contrastive Training for Context Scaling](https://mlsanity.com/paper/2307.03170/) (2023)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references](https://mlsanity.com/paper/1402.4578/) (2014)
- [NLPeer: A Unified Resource for the Computational Study of Peer Review](https://mlsanity.com/paper/2211.06651/) (2022)
- [ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis](https://mlsanity.com/paper/2010.06119/) (2020)
- [MOPRD: A multidisciplinary open peer review dataset](https://mlsanity.com/paper/2212.04972/) (2022)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [An LLM-Enhanced Adversarial Editing System for Lexical Simplification](https://mlsanity.com/paper/2402.14704/) (2024)

## Built on by (most cited first, 30 of 33)

- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows](https://mlsanity.com/paper/2505.19897/) (2025)
- [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025)
- [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026)
- [OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientists](https://mlsanity.com/paper/2511.16931/) (2025)
- [Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluation](https://mlsanity.com/paper/2505.23824/) (2025)
- [Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Papers](https://mlsanity.com/paper/2509.19326/) (2025)
- [Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework](https://mlsanity.com/paper/2508.21422/) (2025)
- [Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem](https://mlsanity.com/paper/2506.08134/) (2025)
- [OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment](https://mlsanity.com/paper/2601.01576/) (2026)
- [Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models](https://mlsanity.com/paper/2512.22145/) (2025)
- [Scaling Laws in Scientific Discovery with AI and Robot Scientists](https://mlsanity.com/paper/2503.22444/) (2025)
- [A Multi-Task Evaluation of LLMs' Processing of Academic Text Input](https://mlsanity.com/paper/2508.11779/) (2025)
- [Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models](https://mlsanity.com/paper/2410.21728/) (2024)
- [NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation](https://mlsanity.com/paper/2509.25179/) (2025)
- [GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses](https://mlsanity.com/paper/2604.11924/) (2026)
- [ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation](https://mlsanity.com/paper/2510.16549/) (2025)
- [A Toolbox for Improving Evolutionary Prompt Search](https://mlsanity.com/paper/2511.05120/) (2025)
- [Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews](https://mlsanity.com/paper/2604.19502/) (2026)
- [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026)
- [A Hierarchical Framework for Measuring Scientific Paper Innovation via Large Language Models](https://mlsanity.com/paper/2504.14620/) (2025)
- [FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research](https://mlsanity.com/paper/2602.14285/) (2026)
- [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026)
- [Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review](https://mlsanity.com/paper/2606.12716/) (2026)
- [LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges](https://mlsanity.com/paper/2606.25057/) (2026)
- [CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review](https://mlsanity.com/paper/2609.05227/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinement](https://mlsanity.com/paper/2512.24754/) (2025)
- [From peer review nuances to best practices](https://mlsanity.com/paper/2607.22681/) (2026)

## Abstract (arXiv)

In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.