# Extending the Frontier of ChatGPT: Code Generation and Debugging

arXiv: https://arxiv.org/abs/2307.08260 · HTML: https://mlsanity.com/paper/2307.08260/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021)
- [Conversational Automated Program Repair](https://mlsanity.com/paper/2301.13246/) (2023)
- [Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs](https://mlsanity.com/paper/2111.03922/) (2021)
- [Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair](https://mlsanity.com/paper/2008.02944/) (2020)
- [DEAR: A Novel Deep Learning-based Approach for Automated Program Repair](https://mlsanity.com/paper/2205.01859/) (2022)
- [KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair](https://mlsanity.com/paper/2302.01857/) (2023)
- [Flood Prediction Using Machine Learning Models](https://mlsanity.com/paper/2208.01234/) (2022)
- [An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode](https://mlsanity.com/paper/2208.08603/) (2022)

## Built on by (most cited first, 9 of 9)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [Learning to Ask: When LLM Agents Meet Unclear Instruction](https://mlsanity.com/paper/2409.00557/) (2024)
- [Competition-Level Problems are Effective LLM Evaluators](https://mlsanity.com/paper/2312.02143/) (2023)
- [LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving](https://mlsanity.com/paper/2502.04355/) (2025)
- [Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for Bangla and Sylheti](https://mlsanity.com/paper/2310.10935/) (2023)
- ["If You're Very Clever, No One Knows You've Used It": The Social Dynamics of Developing Generative AI Literacy in the Workplace](https://mlsanity.com/paper/2602.01386/) (2026)
- [Towards Adaptive Software Agents for Debugging](https://mlsanity.com/paper/2504.18316/) (2025)
- [ChatGPT vs. DeepSeek: A Comparative Study on AI-Based Code Generation](https://mlsanity.com/paper/2502.18467/) (2025)

## Abstract (arXiv)

Large-scale language models (LLMs) have emerged as a groundbreaking innovation in the realm of question-answering and conversational agents. These models, leveraging different deep learning architectures such as Transformers, are trained on vast corpora to predict sentences based on given queries. Among these LLMs, ChatGPT, developed by OpenAI, has ushered in a new era by utilizing artificial intelligence (AI) to tackle diverse problem domains, ranging from composing essays and biographies to solving intricate mathematical integrals. The versatile applications enabled by ChatGPT offer immense value to users. However, assessing the performance of ChatGPT's output poses a challenge, particularly in scenarios where queries lack clear objective criteria for correctness. For instance, evaluating the quality of generated essays becomes arduous and relies heavily on manual labor, in stark contrast to evaluating solutions to well-defined, closed-ended questions such as mathematical problems. This research paper delves into the efficacy of ChatGPT in solving programming problems, examining both the correctness and the efficiency of its solution in terms of time and memory complexity. The research reveals a commendable overall success rate of 71.875%, denoting the proportion of problems for which ChatGPT was able to provide correct solutions that successfully satisfied all the test cases present in Leetcode. It exhibits strengths in structured problems and shows a linear correlation between its success rate and problem acceptance rates. However, it struggles to improve solutions based on feedback, pointing to potential shortcomings in debugging tasks. These findings provide a compact yet insightful glimpse into ChatGPT's capabilities and areas for improvement.