# Evaluating AIGC Detectors on Code Content

arXiv: https://arxiv.org/abs/2304.05193 · HTML: https://mlsanity.com/paper/2304.05193/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2023.

## Built on by (most cited first, 12 of 12)

- [RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://mlsanity.com/paper/2405.07940/) (2024)
- [DF40: Toward Next-Generation Deepfake Detection](https://mlsanity.com/paper/2406.13495/) (2024)
- [Testing of Detection Tools for AI-Generated Text](https://mlsanity.com/paper/2306.15666/) (2023)
- [A Survey on Detection of LLMs-Generated Content](https://mlsanity.com/paper/2310.15654/) (2023)
- [Zero-Shot Detection of Machine-Generated Codes](https://mlsanity.com/paper/2310.05103/) (2023)
- [An Empirical Study of AI Generated Text Detection Tools](https://mlsanity.com/paper/2310.01423/) (2023)
- [ChatGPT Code Detection: Techniques for Uncovering the Source of Code](https://mlsanity.com/paper/2405.15512/) (2024)
- [AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection](https://mlsanity.com/paper/2412.16594/) (2024)
- [Detection of LLM-Paraphrased Code and Identification of the Responsible LLM Using Coding Style Features](https://mlsanity.com/paper/2502.17749/) (2025)
- [Who Writes What: Unveiling the Impact of Author Roles on AI-generated Text Detection](https://mlsanity.com/paper/2502.12611/) (2025)
- [MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios](https://mlsanity.com/paper/2507.21693/) (2025)
- [IndicDetect: Evaluating Cross-Lingual LLM-Generated Text Detection for Hindi, Telugu, and Tamil](https://mlsanity.com/paper/2608.29919/) (2026)

## Abstract (arXiv)

Artificial Intelligence Generated Content (AIGC) has garnered considerable attention for its impressive performance, with ChatGPT emerging as a leading AIGC model that produces high-quality responses across various applications, including software development and maintenance. Despite its potential, the misuse of ChatGPT poses significant concerns, especially in education and safetycritical domains. Numerous AIGC detectors have been developed and evaluated on natural language data. However, their performance on code-related content generated by ChatGPT remains unexplored. To fill this gap, in this paper, we present the first empirical study on evaluating existing AIGC detectors in the software domain. We created a comprehensive dataset including 492.5K samples comprising code-related content produced by ChatGPT, encompassing popular software activities like Q&A (115K), code summarization (126K), and code generation (226.5K). We evaluated six AIGC detectors, including three commercial and three open-source solutions, assessing their performance on this dataset. Additionally, we conducted a human study to understand human detection capabilities and compare them with the existing AIGC detectors. Our results indicate that AIGC detectors demonstrate lower performance on code-related data compared to natural language data. Fine-tuning can enhance detector performance, especially for content within the same domain; but generalization remains a challenge. The human evaluation reveals that detection by humans is quite challenging.