# Can Large Language Models Be Trusted Paper Reviewers? A Feasibility Study

arXiv: https://arxiv.org/abs/2506.17311 · HTML: https://mlsanity.com/paper/2506.17311/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2025.

## Built on by (most cited first, 4 of 4)

- [A Measurement Study of Model Context Protocol Ecosystem](https://mlsanity.com/paper/2509.25292/) (2025)
- [From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation](https://mlsanity.com/paper/2603.17588/) (2026)
- [Before Humans Join the Team: Diagnosing Coordination Failures in Healthcare Robot Team Simulation](https://mlsanity.com/paper/2508.04691/) (2025)
- [GraphReview: Scientific Paper Evaluation via LLM-based Graph Evidence Expansion](https://mlsanity.com/paper/2605.27204/) (2026)

## Abstract (arXiv)

Academic paper review typically requires substantial time, expertise, and human resources. Large Language Models (LLMs) present a promising method for automating the review process due to their extensive training data, broad knowledge base, and relatively low usage cost. This work explores the feasibility of using LLMs for academic paper review by proposing an automated review system. The system integrates Retrieval Augmented Generation (RAG), the AutoGen multi-agent system, and Chain-of-Thought prompting to support tasks such as format checking, standardized evaluation, comment generation, and scoring. Experiments conducted on 290 submissions from the WASA 2024 conference using GPT-4o show that LLM-based review significantly reduces review time (average 2.48 hours) and cost (average 104.28 USD). However, the similarity between LLM-selected papers and actual accepted papers remains low (average 38.6%), indicating issues such as hallucination, lack of independent judgment, and retrieval preferences. Therefore, it is recommended to use LLMs as assistive tools to support human reviewers, rather than to replace them.