# Limits of LLM Text Detectors in Education

arXiv: https://arxiv.org/abs/2508.08096 · HTML: https://mlsanity.com/paper/2508.08096/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [Parsing Argumentation Structures in Persuasive Essays](https://mlsanity.com/paper/1604.07370/) (2016)
- [MGTBench: Benchmarking Machine-Generated Text Detection](https://mlsanity.com/paper/2303.14822/) (2023)
- [Ghostbuster: Detecting Text Ghostwritten by Large Language Models](https://mlsanity.com/paper/2305.15047/) (2023)
- [SeqXGPT: Sentence-Level AI-Generated Text Detection](https://mlsanity.com/paper/2310.08903/) (2023)
- [LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected?](https://mlsanity.com/paper/2401.05952/) (2024)
- [Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases](https://mlsanity.com/paper/2307.07411/) (2023)
- [Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT](https://mlsanity.com/paper/2308.05341/) (2023)
- [Generative AI in Higher Education: Seeing ChatGPT Through Universities' Policies, Resources, and Guidelines](https://mlsanity.com/paper/2312.05235/) (2023)
- [Enhancing Robustness of LLM-Synthetic Text Detectors for Academic Writing: A Comprehensive Analysis](https://mlsanity.com/paper/2401.08046/) (2024)
- [I don't trust you (anymore)! -- The effect of students' LLM use on Lecturer-Student-Trust in Higher Education](https://mlsanity.com/paper/2406.14871/) (2024)

## Built on by (most cited first, 5 of 5)

- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Text](https://mlsanity.com/paper/2509.21269/) (2025)
- [emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity](https://mlsanity.com/paper/2607.19848/) (2026)
- [Non-Parametric Machine Text Detection via Multi-View Gaussian Processes](https://mlsanity.com/paper/2606.14060/) (2026)
- [STEB: Style Text Embedding Benchmark](https://mlsanity.com/paper/2606.31741/) (2026)

## Abstract (arXiv)

Students increasingly use the assistance of large language models (LLMs) in their academic writing. While slight assistance (e.g., grammar and style correction, as well as feedback) is permitted under most institutional policies, it is usually forbidden to offload entire writing tasks to LLMs. Unfortunately, current approaches to LLM-generated text detection predominantly assume a binary distinction between human-written and LLM-generated text, ignoring the breadth of realistic human-AI collaboration practices and limiting the validity of detection systems for educational assessment. In this paper, we propose a contribution-aware evaluation framework for LLM-based detection systems in education. We introduce a scale of eight student contribution levels that model realistic writing scenarios ranging from fully human-written texts to LLM-assisted revisions to fully LLM-generated and adversarially humanized texts. Institutional policies regarding LLM use can then be translated to thresholds of acceptable LLM assistance on this scale. We further present Generative Essay Detection in Education (GEDE), a novel benchmark dataset comprising more than 900 human-written and over 12,500 generated essays across 886 tasks and all contribution levels. Using this benchmark, we conduct a systematic evaluation of four state-of-the-art zero-shot and supervised detection methods across policy boundaries, contribution levels, generative models, out-of-distribution data, and text length. We show that most detectors struggle to accurately classify texts at intermediate student contribution levels, in particular LLM-assisted revisions of human-written texts. Such errors pose a substantial risk of false accusations, indicating that current text detection systems are (still) unsuitable to reliably support the enforcement of institutional policies regarding LLM assistance in education.