# Analysis of ChatGPT on Source Code

arXiv: https://arxiv.org/abs/2306.00597 · HTML: https://mlsanity.com/paper/2306.00597/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 6 of 6)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [ChatGPT is not all you need. A State of the Art Review of large Generative AI models](https://mlsanity.com/paper/2301.04655/) (2023)
- [Memory Augmented Large Language Models are Computationally Universal](https://mlsanity.com/paper/2301.04589/) (2023)
- [A Glimpse in ChatGPT Capabilities and its impact for AI research](https://mlsanity.com/paper/2305.06087/) (2023)

## Built on by (most cited first, 12 of 12)

- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking](https://mlsanity.com/paper/2307.16806/) (2023)
- [Coding by Design: GPT-4 empowers Agile Model Driven Development](https://mlsanity.com/paper/2310.04304/) (2023)
- [Can LLMs be Fooled? Investigating Vulnerabilities in LLMs](https://mlsanity.com/paper/2407.20529/) (2024)
- [Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3](https://mlsanity.com/paper/2504.16027/) (2025)
- [LLM Interactive Optimization of Open Source Python Libraries -- Case Studies and Generalization](https://mlsanity.com/paper/2312.14949/) (2023)
- [CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM Code Assistants](https://mlsanity.com/paper/2404.09066/) (2024)
- [SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://mlsanity.com/paper/2502.18468/) (2025)
- [Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models](https://mlsanity.com/paper/2509.07010/) (2025)
- [LLM as a code generator in Agile Model Driven Development](https://mlsanity.com/paper/2410.18489/) (2024)
- [Can We Make Code Green? Understanding Trade-Offs in LLMs vs. Human Code Optimizations](https://mlsanity.com/paper/2503.20126/) (2025)
- [HyperGraphOS: A Meta Operating System for Science and Engineering](https://mlsanity.com/paper/2412.04923/) (2024)

## Abstract (arXiv)

This paper explores the use of Large Language Models (LLMs) and in particular ChatGPT in programming, source code analysis, and code generation. LLMs and ChatGPT are built using machine learning and artificial intelligence techniques, and they offer several benefits to developers and programmers. While these models can save time and provide highly accurate results, they are not yet advanced enough to replace human programmers entirely. The paper investigates the potential applications of LLMs and ChatGPT in various areas, such as code creation, code documentation, bug detection, refactoring, and more. The paper also suggests that the usage of LLMs and ChatGPT is expected to increase in the future as they offer unparalleled benefits to the programming community.