# Testing of Detection Tools for AI-Generated Text

arXiv: https://arxiv.org/abs/2306.15666 · HTML: https://mlsanity.com/paper/2306.15666/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [On the Possibilities of AI-Generated Text Detection](https://mlsanity.com/paper/2304.04736/) (2023)
- [Will ChatGPT get you caught? Rethinking of Plagiarism Detection](https://mlsanity.com/paper/2302.04335/) (2023)
- [AI vs. Human -- Differentiation Analysis of Scientific Content Generation](https://mlsanity.com/paper/2301.10416/) (2023)
- [To ChatGPT, or not to ChatGPT: That is the question!](https://mlsanity.com/paper/2304.01487/) (2023)
- [A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications](https://mlsanity.com/paper/2202.02013/) (2022)
- [Evaluating AIGC Detectors on Code Content](https://mlsanity.com/paper/2304.05193/) (2023)
- [Testing of Support Tools for Plagiarism Detection](https://mlsanity.com/paper/2002.04279/) (2020)

## Built on by (most cited first, 30 of 90)

- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [A Statistical Framework of Watermarks for Large Language Models: Pivot, Detection Efficiency and Optimal Rules](https://mlsanity.com/paper/2404.01245/) (2024)
- [Can Unconfident LLM Annotations Be Used for Confident Conclusions?](https://mlsanity.com/paper/2408.15204/) (2024)
- [Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices](https://mlsanity.com/paper/2403.12503/) (2024)
- [Technical Report on the Pangram AI-Generated Text Classifier](https://mlsanity.com/paper/2402.14873/) (2024)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods](https://mlsanity.com/paper/2406.15583/) (2024)
- [AI-generated text boundary detection with RoFT](https://mlsanity.com/paper/2311.08349/) (2023)
- [DAMAGE: Detecting Adversarially Modified AI Generated Text](https://mlsanity.com/paper/2501.03437/) (2025)
- [Human-LLM Coevolution: Evidence from Academic Writing](https://mlsanity.com/paper/2502.09606/) (2025)
- [Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text](https://mlsanity.com/paper/2403.05750/) (2024)
- [GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education](https://mlsanity.com/paper/2403.19148/) (2024)
- [Generative AI Literacy: Twelve Defining Competencies](https://mlsanity.com/paper/2412.12107/) (2024)
- [IMGTB: A Framework for Machine-Generated Text Detection Benchmarking](https://mlsanity.com/paper/2311.12574/) (2023)
- [The AI Assessment Scale (AIAS): A Framework for Ethical Integration of Generative AI in Educational Assessment](https://mlsanity.com/paper/2312.07086/) (2023)
- [Generative AI in Higher Education: Seeing ChatGPT Through Universities' Policies, Resources, and Guidelines](https://mlsanity.com/paper/2312.05235/) (2023)
- [GenAI Mirage: The Impostor Bias and the Deepfake Detection Challenge in the Era of Artificial Illusions](https://mlsanity.com/paper/2312.16220/) (2023)
- [Developing Story: Case Studies of Generative AI's Use in Journalism](https://mlsanity.com/paper/2406.13706/) (2024)
- [An Empirical Study of AI Generated Text Detection Tools](https://mlsanity.com/paper/2310.01423/) (2023)
- [The Adoption and Efficacy of Large Language Models: Evidence From Consumer Complaints in the Financial Industry](https://mlsanity.com/paper/2311.16466/) (2023)
- ['Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews](https://mlsanity.com/paper/2410.09770/) (2024)
- [U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AI](https://mlsanity.com/paper/2406.15386/) (2024)
- [GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge](https://mlsanity.com/paper/2412.18274/) (2024)
- [The AI Assessment Scale (AIAS) in action: A pilot implementation of GenAI supported assessment- A Preprint](https://mlsanity.com/paper/2403.14692/) (2024)
- [ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale](https://mlsanity.com/paper/2404.02990/) (2024)
- [Deepfakes and Higher Education: A Research Agenda and Scoping Review of Synthetic Media](https://mlsanity.com/paper/2404.15601/) (2024)
- [Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity](https://mlsanity.com/paper/2407.13105/) (2024)

## Abstract (arXiv)

Recent advances in generative pre-trained transformer large language models have emphasised the potential risks of unfair use of artificial intelligence (AI) generated content in an academic environment and intensified efforts in searching for solutions to detect such content. The paper examines the general functionality of detection tools for artificial intelligence generated text and evaluates them based on accuracy and error type analysis. Specifically, the study seeks to answer research questions about whether existing detection tools can reliably differentiate between human-written text and ChatGPT-generated text, and whether machine translation and content obfuscation techniques affect the detection of AI-generated text. The research covers 12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck) that are widely used in the academic setting. The researchers conclude that the available detection tools are neither accurate nor reliable and have a main bias towards classifying the output as human-written rather than detecting AI-generated text. Furthermore, content obfuscation techniques significantly worsen the performance of tools. The study makes several significant contributions. First, it summarises up-to-date similar scientific and non-scientific efforts in the field. Second, it presents the result of one of the most comprehensive tests conducted so far, based on a rigorous research methodology, an original document set, and a broad coverage of tools. Third, it discusses the implications and drawbacks of using detection tools for AI-generated text in academic settings.