# Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI

arXiv: https://arxiv.org/abs/2604.19578 · HTML: https://mlsanity.com/paper/2604.19578/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates](https://mlsanity.com/paper/2405.02150/) (2024)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis](https://mlsanity.com/paper/2010.06119/) (2020)
- [Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions](https://mlsanity.com/paper/2406.05688/) (2024)
- [GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study](https://mlsanity.com/paper/2307.05492/) (2023)
- [Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis](https://mlsanity.com/paper/2407.12857/) (2024)
- [Is ChatGPT Transforming Academics' Writing Style?](https://mlsanity.com/paper/2404.08627/) (2024)
- [ReviewEval: An Evaluation Framework for AI-Generated Reviews](https://mlsanity.com/paper/2502.11736/) (2025)
- [An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process](https://mlsanity.com/paper/2010.05137/) (2020)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [AI vs. Human -- Differentiation Analysis of Scientific Content Generation](https://mlsanity.com/paper/2301.10416/) (2023)
- [Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review](https://mlsanity.com/paper/2502.19614/) (2025)
- [Automatic Analysis of Substantiation in Scientific Peer Reviews](https://mlsanity.com/paper/2311.11967/) (2023)
- [Can ChatGPT evaluate research quality?](https://mlsanity.com/paper/2402.05519/) (2024)
- [Calibrating "Cheap Signals" in Peer Review without a Prior](https://mlsanity.com/paper/2312.07269/) (2023)

## Built on by (most cited first, 1 of 1)

- [PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers](https://mlsanity.com/paper/2605.26730/) (2026)

## Abstract (arXiv)

With the rapid advancement of Large Language Models (LLMs), the academic community has faced unprecedented disruptions, particularly in the realm of academic communication. The primary function of peer review is improving the quality of academic manuscripts, such as clarity, originality and other evaluation aspects. Although prior studies suggest that LLMs are beginning to influence peer review, it remains unclear whether they are altering its core evaluative functions. Moreover, the extent to which LLMs affect the linguistic form, evaluative focus, and recommendation-related signals of peer-review reports has yet to be systematically examined. In this study, we examine the changes in peer review reports for academic articles following the emergence of LLMs, emphasizing variations at fine-grained level. Specifically, we investigate linguistic features such as the length and complexity of words and sentences in review comments, while also automatically annotating the evaluation aspects of individual review sentences. We also use a maximum likelihood estimation method, previously established, to identify review reports that potentially have modified or generated by LLMs. Finally, we assess the impact of evaluation aspects mentioned in LLM-assisted review reports on the informativeness of recommendation for paper decision-making. The results indicate that following the emergence of LLMs, peer review texts have become longer and more fluent, with increased emphasis on summaries and surface-level clarity, as well as more standardized linguistic patterns, particularly reviewers with lower confidence score. At the same time, attention to deeper evaluative dimensions, such as originality, replicability, and nuanced critical reasoning, has declined.