# GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study

arXiv: https://arxiv.org/abs/2307.05492 · HTML: https://mlsanity.com/paper/2307.05492/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism](https://mlsanity.com/paper/2110.14802/) (2021)
- [CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis](https://mlsanity.com/paper/2211.14286/) (2022)

## Built on by (most cited first, 30 of 40)

- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions](https://mlsanity.com/paper/2406.05688/) (2024)
- [AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews](https://mlsanity.com/paper/2408.10365/) (2024)
- [Automated Focused Feedback Generation for Scientific Writing Assistance](https://mlsanity.com/paper/2405.20477/) (2024)
- [ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews](https://mlsanity.com/paper/2503.08506/) (2025)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [ReviewEval: An Evaluation Framework for AI-Generated Reviews](https://mlsanity.com/paper/2502.11736/) (2025)
- [Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews](https://mlsanity.com/paper/2502.17086/) (2025)
- [Benchmarking LLMs' Judgments with No Gold Standard](https://mlsanity.com/paper/2411.07127/) (2024)
- [Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks](https://mlsanity.com/paper/2506.11113/) (2025)
- [Generative Adversarial Reviews: When LLMs Become the Critic](https://mlsanity.com/paper/2412.10415/) (2024)
- [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [Systematic Task Exploration with LLMs: A Study in Citation Text Generation](https://mlsanity.com/paper/2407.04046/) (2024)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers](https://mlsanity.com/paper/2505.16330/) (2025)
- [RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation](https://mlsanity.com/paper/2603.09723/) (2026)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [A Multi-Task Evaluation of LLMs' Processing of Academic Text Input](https://mlsanity.com/paper/2508.11779/) (2025)
- [Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions](https://mlsanity.com/paper/2508.13214/) (2025)
- [From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent](https://mlsanity.com/paper/2606.13349/) (2026)
- [Multi-Agent LLMs for Generating Research Limitations](https://mlsanity.com/paper/2601.11578/) (2025)
- [APRES: An Agentic Paper Revision and Evaluation System](https://mlsanity.com/paper/2603.03142/) (2026)
- [Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews](https://mlsanity.com/paper/2604.19502/) (2026)
- [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026)
- [Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Research](https://mlsanity.com/paper/2312.03497/) (2023)
- [EchoReview: Learning Peer Review from the Echoes of Scientific Citations](https://mlsanity.com/paper/2602.00733/) (2026)

## Abstract (arXiv)

In this pilot study, we investigate the use of GPT4 to assist in the peer-review process. Our key hypothesis was that GPT-generated reviews could achieve comparable helpfulness to human reviewers. By comparing reviews generated by both human reviewers and GPT models for academic papers submitted to a major machine learning conference, we provide initial evidence that artificial intelligence can contribute effectively to the peer-review process. We also perform robustness experiments with inserted errors to understand which parts of the paper the model tends to focus on. Our findings open new avenues for leveraging machine learning tools to address resource constraints in peer review. The results also shed light on potential enhancements to the review process and lay the groundwork for further research on scaling oversight in a domain where human-feedback is increasingly a scarce resource.