mlsanityData to 14 Oct 2026

GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study

arXiv 2307.05492 ↗First posted 16 Jun 2023cs.HC cs.AI cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

In this pilot study, we investigate the use of GPT4 to assist in the peer-review process. Our key hypothesis was that GPT-generated reviews could achieve comparable helpfulness to human reviewers. By comparing reviews generated by both human reviewers and GPT models for academic papers submitted to a major machine learning conference, we provide initial evidence that artificial intelligence can contribute effectively to the peer-review process. We also perform robustness experiments with inserted errors to understand which parts of the paper the model tends to focus on. Our findings open new avenues for leveraging machine learning tools to address resource constraints in peer review. The results also shed light on potential enhancements to the review process and lay the groundwork for further research on scaling oversight in a domain where human-feedback is increasingly a scarce resource.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 5 of its 22 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

5 of 22 references.

References · 5

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  4. 2021You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanismcited by 19
  5. 2022CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 12; 2026: 17.

40 papers.

Sort:

By area:

  1. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  2. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  3. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  4. 2025Large language models for automated scholarly paper review: A surveycited by 61
  5. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 46
  6. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 44
  7. 2024Automated Focused Feedback Generation for Scientific Writing Assistancecited by 33
  8. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 32
  9. 2024What Can Natural Language Processing Do for Peer Review?cited by 25
  10. 2025ReviewEval: An Evaluation Framework for AI-Generated Reviewscited by 25
  11. 2025Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviewscited by 24
  12. 2024Benchmarking LLMs' Judgments with No Gold Standardcited by 19
28 more
  1. 2025Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attackscited by 16
  2. 2024Generative Adversarial Reviews: When LLMs Become the Criticcited by 12
  3. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  4. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automationcited by 10
  5. 2024Systematic Task Exploration with LLMs: A Study in Citation Text Generationcited by 8
  6. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  7. 2025SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Paperscited by 8
  8. 2026RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generationcited by 6
  9. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  10. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  11. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  12. 2026From Passive Generation to Investigation: A Proactive Scientific Peer Review Agentcited by 4
  13. 2025Multi-Agent LLMs for Generating Research Limitationscited by 3
  14. 2026APRES: An Agentic Paper Revision and Evaluation Systemcited by 3
  15. 2026Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviewscited by 3
  16. 2026When AI reviews science: Can we trust the referee?cited by 2
  17. 2023Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Researchcited by 1
  18. 2026EchoReview: Learning Peer Review from the Echoes of Scientific Citationscited by 1
  19. 2026Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AIcited by 1
  20. 2026EGTR-Review: Efficient Evidence-Grounded Scientific Peer Review Generation via Multi-Agent Teacher Distillationcited by 1
  21. 2026Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Reviewcited by 1
  22. 2026LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challengescited by 1
  23. 2026AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Qualitycited by 1
  24. 2026Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Rankingcited by 0
  25. 2026Beyond Detection: Governing GenAI in Academic Peer Review as a Sociotechnical Challengecited by 0
  26. 2026Towards a new paradigm of scientific discovery with socialized artificial intelligencecited by 0
  27. 2026InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttalcited by 0
  28. 2026Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Reviewcited by 0