# AI-assisted coding: Experiments with GPT-4

arXiv: https://arxiv.org/abs/2304.13187 · HTML: https://mlsanity.com/paper/2304.13187/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 31 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 5 of 5)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models](https://mlsanity.com/paper/2209.08141/) (2022)

## Built on by (most cited first, 30 of 43)

- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge](https://mlsanity.com/paper/2311.08166/) (2023)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [Understanding the Capabilities of Large Language Models for Automated Planning](https://mlsanity.com/paper/2305.16151/) (2023)
- [TypeFly: Flying Drones with Large Language Model](https://mlsanity.com/paper/2312.14950/) (2023)
- [CodeMirage: Hallucinations in Code Generated by Large Language Models](https://mlsanity.com/paper/2408.08333/) (2024)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [Quantum Many-Body Physics Calculations with Large Language Models](https://mlsanity.com/paper/2403.03154/) (2024)
- [Supersonic: Learning to Generate Source Code Optimizations in C/C++](https://mlsanity.com/paper/2309.14846/) (2023)
- [Can Large Language Models Find And Fix Vulnerable Software?](https://mlsanity.com/paper/2308.10345/) (2023)
- [MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimization](https://mlsanity.com/paper/2503.16874/) (2025)
- [Autonomous Code Evolution Meets NP-Completeness](https://mlsanity.com/paper/2509.07367/) (2025)
- [Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Models](https://mlsanity.com/paper/2307.14349/) (2023)
- [What is it for a Machine Learning Model to Have a Capability?](https://mlsanity.com/paper/2405.08989/) (2024)
- [MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2503.14340/) (2025)
- [LLMs for Coding and Robotics Education](https://mlsanity.com/paper/2402.06116/) (2024)
- [How should the advent of large language models affect the practice of science?](https://mlsanity.com/paper/2312.03759/) (2023)
- [DCA-Bench: A Benchmark for Dataset Curation Agents](https://mlsanity.com/paper/2406.07275/) (2024)
- [GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science](https://mlsanity.com/paper/2312.03769/) (2023)
- [From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow](https://mlsanity.com/paper/2509.12443/) (2025)
- [MAPS: Multi-Agent Personality Shaping for Collaborative Reasoning](https://mlsanity.com/paper/2503.16905/) (2025)
- [Towards Automated Data Sciences with Natural Language and SageCopilot: Practices and Lessons Learned](https://mlsanity.com/paper/2407.21040/) (2024)
- [Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks](https://mlsanity.com/paper/2410.14766/) (2024)
- [On Simulation-Guided LLM-based Code Generation for Safe Autonomous Driving Software](https://mlsanity.com/paper/2504.02141/) (2025)
- [Evaluating GPT's Programming Capability through CodeWars' Katas](https://mlsanity.com/paper/2306.01784/) (2023)
- [ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs](https://mlsanity.com/paper/2401.14279/) (2024)

## Abstract (arXiv)

Artificial intelligence (AI) tools based on large language models have acheived human-level performance on some computer programming tasks. We report several experiments using GPT-4 to generate computer code. These experiments demonstrate that AI code generation using the current generation of tools, while powerful, requires substantial human validation to ensure accurate performance. We also demonstrate that GPT-4 refactoring of existing code can significantly improve that code along several established metrics for code quality, and we show that GPT-4 can generate tests with substantial coverage, but that many of the tests fail when applied to the associated code. These findings suggest that while AI coding tools are very powerful, they still require humans in the loop to ensure validity and accuracy of the results.