# Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content

arXiv: https://arxiv.org/abs/2607.20446 · HTML: https://mlsanity.com/paper/2607.20446/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 12 of 12)

- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- [Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education](https://mlsanity.com/paper/2405.02985/) (2024)
- [Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text](https://mlsanity.com/paper/2501.03212/) (2025)
- [Is The Watermarking Of LLM-Generated Code Robust?](https://mlsanity.com/paper/2403.17983/) (2024)
- [LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education](https://mlsanity.com/paper/2411.10455/) (2024)
- [Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students](https://mlsanity.com/paper/2410.09193/) (2024)
- [Workshop Paper: An empirical study to understand how students use ChatGPT for writing essays and how it affects their ownership](https://mlsanity.com/paper/2405.13890/) (2024)
- [Are You Human? An Adversarial Benchmark to Expose LLMs](https://mlsanity.com/paper/2410.09569/) (2024)

## Built on by (most cited first, 1 of 1)

- [Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis](https://mlsanity.com/paper/2609.34691/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are increasingly used by students to generate natural language responses and program code, there is growing interest in whether LLMs themselves can be used to distinguish AI-generated work from human-authored submissions. In this paper, we investigate the extent to which LLMs can detect their own generated content across multiple educational task types, including programming exercises, reflective writing, and short-answer questions. Using authentic student responses and multiple variants of LLM-generated answers, we evaluate detection performance under different prompting strategies and output formats. Our study addresses three research questions: (1) how accurately LLMs can identify their own outputs across task domains, (2) how detection effectiveness is influenced by factors such as prompt design, response length, and task type, and (3) what characteristics of LLM-generated responses contribute to successful or failed detection. Our findings show that LLM-based detection is highly task-dependent: detection is substantially more reliable for programming tasks and longer reflective responses, but performs poorly for short-answer questions, where LLMs frequently judge their own outputs as more human-like than authentic student responses. We further find that prompt framing and response verbosity have a pronounced effect on detectability in reflective writing tasks, with relatively minor prompt variations significantly reducing detection accuracy, while programming-related detection is more robust to prompt changes. Together, these results highlight both the potential and the limitations of LLM self-detection in educational settings and suggest caution in relying on LLMs as standalone tools for identifying AI-generated student work.