mlsanityData to 7 Oct 2026

ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

arXiv 2306.00622 ↗First posted 1 Jun 2023cs.CL cs.AI cs.DLcited by 118

Not on the map. Why?

Abstract (arXiv)

Given the rapid ascent of large language models (LLMs), we study the question: (How) can large language models help in reviewing of scientific papers or proposals? We first conduct some pilot studies where we find that (i) GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to identify errors) outperforms prompting to simply write a review. With these insights, we study the use of LLMs (specifically, GPT-4) for three tasks: 1. Identifying errors: We construct 13 short computer science papers each with a deliberately inserted error, and ask the LLM to check for the correctness of these papers. We observe that the LLM finds errors in 7 of them, spanning both mathematical and conceptual errors. 2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist questions in the respective sections of 15 NeurIPS 2022 papers. We find that across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy. 3. Choosing the "better" paper: We generate 10 pairs of abstracts, deliberately designing each pair in such a way that one abstract was clearly superior than the other. The LLM, however, struggled to discern these relatively straightforward distinctions accurately, committing errors in its evaluations for 6 out of the 10 pairs. Based on these experiments, we think that LLMs have a promising use as reviewing assistants for specific reviewing tasks, but not (yet) for complete evaluations of papers or proposals.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 53 of its 92 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

53 of 92 references.

References · 53

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
38 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  5. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  6. 2023How Language Model Hallucinations Can Snowballcited by 310
  7. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  8. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  9. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  10. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  11. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  12. 2021Generating Datasets with Pretrained Language Modelscited by 192
  13. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  14. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 135
  15. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  16. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  17. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  18. 2022Training Trajectories of Language Models Across Scalescited by 80
  19. 2021Can We Automate Scientific Reviewing?cited by 63
  20. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
  21. 2020ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesiscited by 53
  22. 2019Argument Mining for Understanding Peer Reviewscited by 41
  23. 2020Mitigating Manipulation in Peer Review via Randomized Reviewer Assignmentscited by 38
  24. 2022Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Reviewcited by 37
  25. 2021DISAPERE: A Dataset for Discourse Structure in Peer Review Discussionscited by 35
  26. 2023LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluationcited by 33
  27. 2018PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Reviewcited by 30
  28. 2019Paper Matching with Local Fairness Constraintscited by 30
  29. 2019Simple and Effective Paraphrastic Similarity from Parallel Translationscited by 25
  30. 2020Aspect-based Sentiment Analysis of Scientific Reviewscited by 24
  31. 2022Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approachcited by 23
  32. 2020Argument Mining Driven Analysis of Peer-Reviewscited by 16
  33. 2020Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessmentcited by 14
  34. 2023Large Language Models as Corporate Lobbyistscited by 13
  35. 2020Uncovering Latent Biases in Text: Method and Application to Peer Reviewcited by 11
  36. 2023A Gold Standard Dataset for the Reviewer Assignment Problemcited by 11
  37. 2018Loss Functions, Axioms, and Peer Reviewcited by 10
  38. 2022A Dataset on Malicious Paper Bidding in Peer Reviewcited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 25; 2025: 40; 2026: 45.

118 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  3. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 160
  4. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  5. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 112
  6. 2024MARG: Multi-Agent Review Generation for Scientific Paperscited by 98
  7. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  8. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  9. 2024OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviewscited by 63
  10. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  11. 2025Large language models for automated scholarly paper review: A surveycited by 59
  12. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
38 more
  1. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 52
  2. 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 49
  3. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 43
  4. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 42
  5. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  6. 2023GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Studycited by 40
  7. 2025Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025cited by 40
  8. 2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
  9. 2025Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewardscited by 34
  10. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  11. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 28
  12. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
  13. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 25
  14. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  15. 2025The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemscited by 24
  16. 2025ReviewEval: An Evaluation Framework for AI-Generated Reviewscited by 22
  17. 2024Benchmarking LLMs' Judgments with No Gold Standardcited by 19
  18. 2025Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Reviewcited by 18
  19. 2025Detecting LLM-Generated Peer Reviewscited by 18
  20. 2024LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Paperscited by 16
  21. 2023Reproducibility in Machine Learning-Driven Researchcited by 15
  22. 2024Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experimentcited by 15
  23. 2025How Far Are AI Scientists from Changing the World?cited by 15
  24. 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 15
  25. 2023Improving Interpersonal Communication by Simulating Audiences with Language Modelscited by 14
  26. 2024Eliciting Informative Text Evaluations with Large Language Modelscited by 14
  27. 2025Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attackscited by 14
  28. 2024Generative Adversarial Reviews: When LLMs Become the Criticcited by 12
  29. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12
  30. 2025From Reproduction to Replication: Evaluating Research Agents with Progressive Code Maskingcited by 12
  31. 2025ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Reviewcited by 12
  32. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 12
  33. 2026ARIS: Autonomous Research via Adversarial Multi-Agent Collaborationcited by 12
  34. 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 12
  35. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  36. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 10
  37. 2025REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learningcited by 9
  38. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9

50 most cited of 118 All 118 (JSON)