# Is this Snippet Written by ChatGPT? An Empirical Study with a CodeBERT-Based Classifier

arXiv: https://arxiv.org/abs/2307.09381 · HTML: https://mlsanity.com/paper/2307.09381/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2023.

## Built on by (most cited first, 6 of 6)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers](https://mlsanity.com/paper/2401.06461/) (2024)
- [CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs](https://mlsanity.com/paper/2506.11059/) (2025)
- [Detection of LLM-Paraphrased Code and Identification of the Responsible LLM Using Coding Style Features](https://mlsanity.com/paper/2502.17749/) (2025)
- [An Exploratory Study on LLM-Generated Code and Comments in Code Repositories](https://mlsanity.com/paper/2607.01867/) (2026)
- [Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities](https://mlsanity.com/paper/2409.01382/) (2024)

## Abstract (arXiv)

Since its launch in November 2022, ChatGPT has gained popularity among users, especially programmers who use it as a tool to solve development problems. However, while offering a practical solution to programming problems, ChatGPT should be mainly used as a supporting tool (e.g., in software education) rather than as a replacement for the human being. Thus, detecting automatically generated source code by ChatGPT is necessary, and tools for identifying AI-generated content may need to be adapted to work effectively with source code. This paper presents an empirical study to investigate the feasibility of automated identification of AI-generated code snippets, and the factors that influence this ability. To this end, we propose a novel approach called GPTSniffer, which builds on top of CodeBERT to detect source code written by AI. The results show that GPTSniffer can accurately classify whether code is human-written or AI-generated, and outperforms two baselines, GPTZero and OpenAI Text Classifier. Also, the study shows how similar training data or a classification context with paired snippets helps to boost classification performances.