# Are AI Detectors Good Enough? A Survey on Quality of Datasets With Machine-Generated Texts

arXiv: https://arxiv.org/abs/2410.14677 · HTML: https://mlsanity.com/paper/2410.14677/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [TweepFake: about Detecting Deepfake Tweets](https://mlsanity.com/paper/2008.00036/) (2020)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [MGTBench: Benchmarking Machine-Generated Text Detection](https://mlsanity.com/paper/2303.14822/) (2023)
- [OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples](https://mlsanity.com/paper/2307.11729/) (2023)
- [HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus](https://mlsanity.com/paper/2309.02731/) (2023)
- [Overview of AuTexTification at IberLEF 2023: Detection and Attribution of Machine-Generated Text in Multiple Domains](https://mlsanity.com/paper/2309.11285/) (2023)
- [Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian](https://mlsanity.com/paper/2206.01583/) (2022)
- [ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature](https://mlsanity.com/paper/2403.16887/) (2024)
- [INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback](https://mlsanity.com/paper/2305.14282/) (2023)
- [AI vs. Human -- Differentiation Analysis of Scientific Content Generation](https://mlsanity.com/paper/2301.10416/) (2023)
- [SynSciPass: detecting appropriate uses of scientific text generation](https://mlsanity.com/paper/2209.03742/) (2022)

## Built on by (most cited first, 10 of 10)

- [Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders](https://mlsanity.com/paper/2503.03601/) (2025)
- [How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study](https://mlsanity.com/paper/2510.13681/) (2025)
- [GigaCheck: Detecting LLM-generated Content via Object-Centric Span Localization](https://mlsanity.com/paper/2410.23728/) (2024)
- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [Advacheck at GenAI Detection Task 1: AI Detection Powered by Domain-Aware Multi-Tasking](https://mlsanity.com/paper/2411.11736/) (2024)
- [LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Text](https://mlsanity.com/paper/2509.21269/) (2025)
- [Spotlights and Blindspots: Evaluating Machine-Generated Text Detection](https://mlsanity.com/paper/2604.16607/) (2026)
- [RedNote-Vibe: A Dataset for Capturing Temporal Dynamics of AI-Generated Text in Lifestyle Social Media](https://mlsanity.com/paper/2509.22055/) (2025)
- [Ontological Instability and Statistical Amplification: The Paradox of "Humanizing" LLM-Generated Text](https://mlsanity.com/paper/2610.03110/) (2026)
- [Measuring Artifact Reliance in Machine-Generated Text Detectors](https://mlsanity.com/paper/2610.13583/) (2026)

## Abstract (arXiv)

The rapid development of autoregressive Large Language Models (LLMs) has significantly improved the quality of generated texts, necessitating reliable machine-generated text detectors. A huge number of detectors and collections with AI fragments have emerged, and several detection methods even showed recognition quality up to 99.9% according to the target metrics in such collections. However, the quality of such detectors tends to drop dramatically in the wild, posing a question: Are detectors actually highly trustworthy or do their high benchmark scores come from the poor quality of evaluation datasets? In this paper, we emphasise the need for robust and qualitative methods for evaluating generated data to be secure against bias and low generalising ability of future model. We present a systematic review of datasets from competitions dedicated to AI-generated content detection and propose methods for evaluating the quality of datasets containing AI-generated fragments. In addition, we discuss the possibility of using high-quality generated data to achieve two goals: improving the training of detection models and improving the training datasets themselves. Our contribution aims to facilitate a better understanding of the dynamics between human and machine text, which will ultimately support the integrity of information in an increasingly automated world. The code is available at https://github.com/Advacheck-OU/ai-dataset-analysing.