# Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases

arXiv: https://arxiv.org/abs/2307.07411 · HTML: https://mlsanity.com/paper/2307.07411/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 11 of 11)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [The Science of Detecting LLM-Generated Texts](https://mlsanity.com/paper/2303.07205/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Academic Source Code Plagiarism Detection by Measuring Program Behavioural Similarity](https://mlsanity.com/paper/2102.03995/) (2021)

## Built on by (most cited first, 30 of 32)

- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent](https://mlsanity.com/paper/2310.06155/) (2023)
- [Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods](https://mlsanity.com/paper/2406.15583/) (2024)
- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Why Can Large Language Models Generate Correct Chain-of-Thoughts?](https://mlsanity.com/paper/2310.13571/) (2023)
- [The AI Assessment Scale (AIAS): A Framework for Ethical Integration of Generative AI in Educational Assessment](https://mlsanity.com/paper/2312.07086/) (2023)
- [Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems](https://mlsanity.com/paper/2410.18336/) (2024)
- [Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text](https://mlsanity.com/paper/2501.03212/) (2025)
- [Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits](https://mlsanity.com/paper/2403.14791/) (2024)
- [KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis](https://mlsanity.com/paper/2503.00032/) (2025)
- [Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation](https://mlsanity.com/paper/2410.09318/) (2024)
- [Limits of LLM Text Detectors in Education](https://mlsanity.com/paper/2508.08096/) (2025)
- [Evaluating multiple large language models in pediatric ophthalmology](https://mlsanity.com/paper/2311.04368/) (2023)
- [Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education](https://mlsanity.com/paper/2407.17022/) (2024)
- [Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction](https://mlsanity.com/paper/2412.15253/) (2024)
- [Recognising, Anticipating, and Mitigating LLM Pollution of Online Behavioural Research](https://mlsanity.com/paper/2508.01390/) (2025)
- [Faculty Perspectives on the Potential of RAG in Computer Science Higher Education](https://mlsanity.com/paper/2408.01462/) (2024)
- [Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners](https://mlsanity.com/paper/2412.12800/) (2024)
- [The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own](https://mlsanity.com/paper/2503.05760/) (2025)
- [You've Changed: Detecting Modification of Black-Box Large Language Models](https://mlsanity.com/paper/2504.12335/) (2025)
- [An evaluation of LLMs for generating movie reviews: GPT-4o, Gemini-2.0 and DeepSeek-V3](https://mlsanity.com/paper/2506.00312/) (2025)
- [Student Mastery or AI Deception? Analyzing ChatGPT's Assessment Proficiency and Evaluating Detection Strategies](https://mlsanity.com/paper/2311.16292/) (2023)
- [Assessing the Performance of Analog Training for Transfer Learning](https://mlsanity.com/paper/2505.11067/) (2025)
- [Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Prompting](https://mlsanity.com/paper/2408.01423/) (2024)
- [Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts](https://mlsanity.com/paper/2507.03811/) (2025)
- [Evaluating the Performance of AI Text Detectors, Few-Shot and Chain-of-Thought Prompting Using DeepSeek Generated Text](https://mlsanity.com/paper/2507.17944/) (2025)
- [Vibe Learning: Education in the age of AI](https://mlsanity.com/paper/2511.01956/) (2025)
- [Can We Trust LLM Detectors?](https://mlsanity.com/paper/2601.15301/) (2026)

## Abstract (arXiv)

Due to the recent improvements and wide availability of Large Language Models (LLMs), they have posed a serious threat to academic integrity in education. Modern LLM-generated text detectors attempt to combat the problem by offering educators with services to assess whether some text is LLM-generated. In this work, we have collected 124 submissions from computer science students before the creation of ChatGPT. We then generated 40 ChatGPT submissions. We used this data to evaluate eight publicly-available LLM-generated text detectors through the measures of accuracy, false positives, and resilience. The purpose of this work is to inform the community of what LLM-generated text detectors work and which do not, but also to provide insights for educators to better maintain academic integrity in their courses. Our results find that CopyLeaks is the most accurate LLM-generated text detector, GPTKit is the best LLM-generated text detector to reduce false positives, and GLTR is the most resilient LLM-generated text detector. We also express concerns over 52 false positives (of 114 human written submissions) generated by GPTZero. Finally, we note that all LLM-generated text detectors are less accurate with code, other languages (aside from English), and after the use of paraphrasing tools (like QuillBot). Modern detectors are still in need of improvements so that they can offer a full-proof solution to help maintain academic integrity. Further, their usability can be improved by facilitating a smooth API integration, providing clear documentation of their features and the understandability of their model(s), and supporting more commonly used languages.