# DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios

arXiv: https://arxiv.org/abs/2410.23746 · HTML: https://mlsanity.com/paper/2410.23746/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 71 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 46)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [TweepFake: about Detecting Deepfake Tweets](https://mlsanity.com/paper/2008.00036/) (2020)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)

## Built on by (most cited first, 30 of 38)

- [RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns](https://mlsanity.com/paper/2508.13152/) (2025)
- [DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning](https://mlsanity.com/paper/2510.17489/) (2025)
- [DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm](https://mlsanity.com/paper/2509.15550/) (2025)
- [Characterizing LLM-driven Social Network: The Chirper.ai Case](https://mlsanity.com/paper/2504.10286/) (2025)
- [DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models](https://mlsanity.com/paper/2509.14268/) (2025)
- [Who's Your Judge? On the Detectability of LLM-Generated Judgments](https://mlsanity.com/paper/2509.25154/) (2025)
- [Gen-Review: A Large-scale Dataset of AI-Generated (and Human-written) Peer Reviews](https://mlsanity.com/paper/2510.21192/) (2025)
- [Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model](https://mlsanity.com/paper/2604.21223/) (2026)
- [A Comprehensive Dataset for Human vs. AI Generated Text Detection](https://mlsanity.com/paper/2510.22874/) (2025)
- [Pangram 4 Technical Report](https://mlsanity.com/paper/2607.27183/) (2026)
- [IPAD: Inverse Prompt for AI Detection - A Robust and Interpretable LLM-Generated Text Detector](https://mlsanity.com/paper/2502.15902/) (2025)
- [LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Text](https://mlsanity.com/paper/2509.21269/) (2025)
- [Advancing Machine-Generated Text Detection from an Easy to Hard Supervision Perspective](https://mlsanity.com/paper/2511.00988/) (2025)
- [Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection](https://mlsanity.com/paper/2602.08031/) (2026)
- [TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices](https://mlsanity.com/paper/2605.31113/) (2026)
- [An Exploratory Study on LLM-Generated Code and Comments in Code Repositories](https://mlsanity.com/paper/2607.01867/) (2026)
- [Black-Box Detection of LLM-Generated Text Using Generalized Jensen-Shannon Divergence](https://mlsanity.com/paper/2510.07500/) (2025)
- [EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection through Diverse Model Ensembles](https://mlsanity.com/paper/2603.27949/) (2026)
- [Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy](https://mlsanity.com/paper/2604.16923/) (2026)
- [DSIPA: Detecting LLM-Generated Texts via Sentiment-Invariant Patterns Divergence Analysis](https://mlsanity.com/paper/2604.26328/) (2026)
- [MELD: Multi-Task Equilibrated Learning Detector for AI-Generated Text](https://mlsanity.com/paper/2605.06903/) (2026)
- [DetectZoo: A Unified Toolkit for AI-Generated Content Detection Across Text, Audio, and Image Modalities](https://mlsanity.com/paper/2606.04205/) (2026)
- [Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection](https://mlsanity.com/paper/2606.06481/) (2026)
- [ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation](https://mlsanity.com/paper/2607.29539/) (2026)
- [Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration](https://mlsanity.com/paper/2608.05741/) (2026)
- [EVIL-Detect for NLPCC 2026 Shared Task 6: LLM-Generated Text Detection](https://mlsanity.com/paper/2608.10698/) (2026)
- [Beyond Perplexity: Character Distribution Signatures and the MDTA Benchmark for AI Text Detection](https://mlsanity.com/paper/2605.01647/) (2026)
- [Steer-to-Detect: Probing Hidden Representations for Detection of LLM-Generated Texts](https://mlsanity.com/paper/2605.12890/) (2026)
- [Multi-Level Contextual Token Relation Modeling for Machine-Generated Text Detection](https://mlsanity.com/paper/2605.16107/) (2026)
- [Hidden Human-Like Nature of Machine-Generated Texts: Theory and Detection Enhancement](https://mlsanity.com/paper/2605.23190/) (2026)

## Abstract (arXiv)

Detecting text generated by large language models (LLMs) is of great recent interest. With zero-shot methods like DetectGPT, detection capabilities have reached impressive levels. However, the reliability of existing detectors in real-world applications remains underexplored. In this study, we present a new benchmark, DetectRL, highlighting that even state-of-the-art (SOTA) detection techniques still underperformed in this task. We collected human-written datasets from domains where LLMs are particularly prone to misuse. Using popular LLMs, we generated data that better aligns with real-world applications. Unlike previous studies, we employed heuristic rules to create adversarial LLM-generated text, simulating various prompts usages, human revisions like word substitutions, and writing noises like spelling mistakes. Our development of DetectRL reveals the strengths and limitations of current SOTA detectors. More importantly, we analyzed the potential impact of writing styles, model types, attack methods, the text lengths, and real-world human writing factors on different types of detectors. We believe DetectRL could serve as an effective benchmark for assessing detectors in real-world scenarios, evolving with advanced attack methods, thus providing more stressful evaluation to drive the development of more efficient detectors. Data and code are publicly available at: https://github.com/NLP2CT/DetectRL.