mlsanityData to 7 Oct 2026

Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews

arXiv 2403.07183 ↗First posted 11 Mar 2024cs.CL cs.AI cs.LGcited by 160

Not on the map. Why?

Abstract (arXiv)

We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Builds on 45 of its 85 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

45 of 85 references.

References · 45

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  4. 2023A Watermark for Large Language Modelscited by 667
  5. 2019Defending Against Neural Fake Newscited by 553
  6. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  7. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 490
  8. 2023Can AI-Generated Text be Reliably Detected?cited by 372
  9. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  10. 2023Robust Distortion-free Watermarks for Language Modelscited by 279
  11. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 278
  12. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 260
33 more
  1. 2023Provable Robust Watermarking for AI-Generated Textcited by 259
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  3. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  4. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  5. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 187
  6. 2023GPT detectors are biased against non-native English writerscited by 175
  7. 2020Automatic Detection of Machine Generated Text: A Critical Surveycited by 160
  8. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  9. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  10. 2023DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Textcited by 127
  11. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  12. 2020TweepFake: about Detecting Deepfake Tweetscited by 117
  13. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  14. 2023Unbiased Watermark for Large Language Modelscited by 100
  15. 2019Real or Fake? Learning to Discriminate Machine from Human Generated Textcited by 98
  16. 2021Algorithmic Monoculture and Social Welfarecited by 98
  17. 2023A Semantic Invariant Robust Watermark for Large Language Modelscited by 97
  18. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  19. 2023On the Possibilities of AI-Generated Text Detectioncited by 88
  20. 2023Testing of Detection Tools for AI-Generated Textcited by 86
  21. 2023Three Bricks to Consolidate Watermarks for Large Language Modelscited by 80
  22. 2023Robust Multi-bit Natural Language Watermarking through Invariant Featurescited by 76
  23. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
  24. 2023A Survey on Detection of LLMs-Generated Contentcited by 70
  25. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  26. 2023Red Teaming Language Model Detectors with Language Modelscited by 55
  27. 2023Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedycited by 45
  28. 2023GPT-Sentinel: Distinguishing Human and ChatGPT Generated Contentcited by 40
  29. 2020Attacking Neural Text Detectorscited by 37
  30. 2022CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learningcited by 32
  31. 2022Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyondcited by 12
  32. 2023New Evaluation Metrics Capture Quality Degradation due to LLM Watermarkingcited by 10
  33. 2023Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectorscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 48; 2026: 72.

160 papers.

Sort:

By area:

  1. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 112
  2. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  3. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 97
  4. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  5. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  6. 2024Delving into LLM-assisted writing in biomedical publications through excess vocabularycited by 75
  7. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  8. 2024Do LLMs write like humans? Variation in grammatical and rhetorical stylescited by 62
  9. 2025Large language models for automated scholarly paper review: A surveycited by 59
  10. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  11. 2025The Widespread Adoption of Large Language Model-Assisted Writing Across Societycited by 50
  12. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
38 more
  1. 2025Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025cited by 40
  2. 2025Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewardscited by 34
  3. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  4. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  5. 2024Is ChatGPT Transforming Academics' Writing Style?cited by 26
  6. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  7. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 24
  8. 2024Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Reviewcited by 19
  9. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  10. 2025LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendationcited by 19
  11. 2024The Impact of Large Language Models in Academia: from Writing to Speakingcited by 18
  12. 2025Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Reviewcited by 18
  13. 2025Detecting LLM-Generated Peer Reviewscited by 18
  14. 2024AIGS: Generating Science from AI-Powered Automated Falsificationcited by 17
  15. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  16. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  17. 2024Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experimentcited by 15
  18. 2025Human-LLM Coevolution: Evidence from Academic Writingcited by 15
  19. 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 15
  20. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 14
  21. 2025ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Reviewcited by 12
  22. 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 12
  23. 2024Evaluating the Predictive Capacity of ChatGPT for Academic Peer Review Outcomes Across Multiple Platformscited by 11
  24. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  25. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  26. 2024Developing Story: Case Studies of Generative AI's Use in Journalismcited by 10
  27. 2024System 2 thinking in OpenAI's o1-preview model: Near-perfect performance on a mathematics examcited by 10
  28. 2025Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectorscited by 10
  29. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 10
  30. 2025In-Context Watermarks for Large Language Modelscited by 9
  31. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  32. 2025AI use in American newspapers is widespread, uneven, and rarely disclosedcited by 9
  33. 2024Examining the Prevalence and Dynamics of AI-Generated Media in Art Subredditscited by 8
  34. 2024'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviewscited by 8
  35. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 8
  36. 2024Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Modelscited by 8
  37. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  38. 2025BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?cited by 8

50 most cited of 160 All 160 (JSON)