# Exploring the Limitations of Detecting Machine-Generated Text

arXiv: https://arxiv.org/abs/2406.11073 · HTML: https://mlsanity.com/paper/2406.11073/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [MGTBench: Benchmarking Machine-Generated Text Detection](https://mlsanity.com/paper/2303.14822/) (2023)
- [OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples](https://mlsanity.com/paper/2307.11729/) (2023)
- [Automatic Detection of Entity-Manipulated Text using Factual Knowledge](https://mlsanity.com/paper/2203.10343/) (2022)
- [Detecting AI Generated Text Based on NLP and Machine Learning Approaches](https://mlsanity.com/paper/2404.10032/) (2024)

## Built on by (most cited first, 15 of 15)

- [People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text](https://mlsanity.com/paper/2501.15654/) (2025)
- [Human-LLM Coevolution: Evidence from Academic Writing](https://mlsanity.com/paper/2502.09606/) (2025)
- [Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors](https://mlsanity.com/paper/2505.24523/) (2025)
- [Linguistic and Embedding-Based Profiling of Texts generated by Humans and Large Language Models](https://mlsanity.com/paper/2507.13614/) (2025)
- [Diversity Boosts AI-Generated Text Detection](https://mlsanity.com/paper/2509.18880/) (2025)
- [Wikipedia in the Era of LLMs: Evolution and Risks](https://mlsanity.com/paper/2503.02879/) (2025)
- [AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software](https://mlsanity.com/paper/2512.18567/) (2025)
- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy](https://mlsanity.com/paper/2603.23146/) (2026)
- [TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices](https://mlsanity.com/paper/2605.31113/) (2026)
- [Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities](https://mlsanity.com/paper/2605.04127/) (2026)
- [A Systematic Analysis of Linguistic Features in AI-Generated Text Detection Across Domains and Models](https://mlsanity.com/paper/2606.04177/) (2026)
- [Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue](https://mlsanity.com/paper/2606.21844/) (2026)
- [Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text](https://mlsanity.com/paper/2608.24780/) (2026)
- [Detecting GPT-Assisted Writing Using Interpretable Stylometric Features](https://mlsanity.com/paper/2609.26687/) (2026)

## Abstract (arXiv)

Recent improvements in the quality of the generations by large language models have spurred research into identifying machine-generated text. Such work often presents high-performing detectors. However, humans and machines can produce text in different styles and domains, yet the performance impact of such on machine generated text detection systems remains unclear. In this paper, we audit the classification performance for detecting machine-generated text by evaluating on texts with varying writing styles. We find that classifiers are highly sensitive to stylistic changes and differences in text complexity, and in some cases degrade entirely to random classifiers. We further find that detection systems are particularly susceptible to misclassify easy-to-read texts while they have high performance for complex texts, leading to concerns about the reliability of detection systems. We recommend that future work attends to stylistic factors and reading difficulty levels of human-written and machine-generated text.