mlsanityData to 14 Oct 2026

People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

arXiv 2501.15654 ↗First posted 26 Jan 2025cs.CL cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts' free-form explanations shows that while they rely heavily on specific lexical clues ('AI vocabulary'), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 26 Jan 2025.

Builds on

29 of 46 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  3. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  4. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  5. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 496
  6. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 287
  9. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 263
  10. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 261
  11. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 190
  12. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
17 more
  1. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 163
  2. 2023MAGE: Machine-generated Text Detection in the Wildcited by 132
  3. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  4. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  5. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  6. 2022Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Textcited by 63
  7. 2023Large Language Models can be Guided to Evade AI-Generated Text Detectioncited by 44
  8. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  9. 2020RoFT: A Tool for Evaluating Human Detection of Machine-Generated Textcited by 36
  10. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  11. 2024Technical Report on the Pangram AI-Generated Text Classifiercited by 34
  12. 2024Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attackcited by 27
  13. 2023AI vs. Human -- Differentiation Analysis of Scientific Content Generationcited by 21
  14. 2025DAMAGE: Detecting Adversarially Modified AI Generated Textcited by 20
  15. 2024Exploring the Limitations of Detecting Machine-Generated Textcited by 15
  16. 2024AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italiancited by 12
  17. 2024LLM Detectors Still Fall Short of Real World: Case of LLM-Generated Short News-Like Postscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 20.

40 papers.

Sort:

By area:

  1. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 58
  2. 2025Idiosyncrasies in Large Language Modelscited by 33
  3. 2025EditLens: Quantifying the Extent of AI Editing in Textcited by 26
  4. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  5. 2025Human-LLM Coevolution: Evidence from Academic Writingcited by 15
  6. 2025AI Safety Should Prioritize the Future of Workcited by 11
  7. 2025AI use in American newspapers is widespread, uneven, and rarely disclosedcited by 10
  8. 2026Can Good Writing Be Generative? Expert-Level AI Writing Emerges through Fine-Tuning on High-Quality Bookscited by 9
  9. 2026StoryScope: Investigating idiosyncrasies in AI fictioncited by 9
  10. 2025Frankentext: Stitching random text fragments into long-form narrativescited by 8
  11. 2025Readers Prefer Outputs of AI Trained on Copyrighted Books over Expert Human Writerscited by 8
  12. 2025Domain Regeneration: How well do LLMs match syntactic properties of text domains?cited by 6
28 more
  1. 2025Measuring AI "Slop" in Textcited by 6
  2. 2025Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativitycited by 5
  3. 2025Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AIcited by 4
  4. 2025LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscationcited by 3
  5. 2025On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?cited by 3
  6. 2026Pangram 4 Technical Reportcited by 3
  7. 2025The AI Memory Gap: Users Misremember What They Created With AI or Withoutcited by 2
  8. 2026An Exploratory Study on LLM-Generated Code and Comments in Code Repositoriescited by 2
  9. 2026Generative AI floods and dilutes the market for bookscited by 2
  10. 2025When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detectioncited by 1
  11. 2026Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Paperscited by 1
  12. 2026Can You Make It Sound Like You? Post-Editing LLM-Generated Text for Personal Stylecited by 1
  13. 2026Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shiftcited by 1
  14. 2026The Perplexity Trap: When Patent Law Makes Human Writing Look Like AIcited by 1
  15. 2025Coreference as an indicator of context scope in multimodal narrativecited by 0
  16. 2025Improving Detection of Watermarked Language Modelscited by 0
  17. 2025Surface Reading LLMs: Synthetic Text and its Stylescited by 0
  18. 2026When LLM Essays Outscore Student Essays: What a Korean Writing Rubric Rewards and Where Readers Disagreecited by 0
  19. 2026Beyond Detection: Rethinking Education in the Age of AI-writingcited by 0
  20. 2026What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experiencecited by 0
  21. 2026Show, Don't TELL: Explainable AI-Generated Text Detectioncited by 0
  22. 2026CASPER in the Machine: Insights into Character Variety in LLM-Generated Storiescited by 0
  23. 2026CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Storiescited by 0
  24. 2026Relational Over-Regularization: Graph-Based AI-Generated Text Detection via Sentence Transition Deviationcited by 0
  25. 2026Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignmentcited by 0
  26. 2026Agents Can Use Base Models to Evade AI Detectioncited by 0
  27. 2026Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysiscited by 0
  28. 2026Calibrating LLM Explanations to Human Decisions with Reinforcement Learningcited by 0