mlsanityData to 14 Oct 2026

AI-assisted coding: Experiments with GPT-4

arXiv 2304.13187 ↗First posted 25 Apr 2023cs.AI cs.SEcited by 43

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) tools based on large language models have acheived human-level performance on some computer programming tasks. We report several experiments using GPT-4 to generate computer code. These experiments demonstrate that AI code generation using the current generation of tools, while powerful, requires substantial human validation to ensure accurate performance. We also demonstrate that GPT-4 refactoring of existing code can significantly improve that code along several established metrics for code quality, and we show that GPT-4 can generate tests with substantial coverage, but that many of the tests fail when applied to the associated code. These findings suggest that while AI coding tools are very powerful, they still require humans in the loop to ensure validity and accuracy of the results.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 5 of its 31 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

5 of 31 references.

References · 5

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2022Psychologically-informed chain-of-thought prompts for metaphor understanding in large language modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 14; 2025: 9; 2026: 4.

43 papers.

Sort:

By area:

  1. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  2. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  3. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  4. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
  5. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  6. 2023MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledgecited by 63
  7. 2023Visual cognition in multimodal large language modelscited by 38
  8. 2023Understanding the Capabilities of Large Language Models for Automated Planningcited by 34
  9. 2023TypeFly: Flying Drones with Large Language Modelcited by 23
  10. 2024CodeMirage: Hallucinations in Code Generated by Large Language Modelscited by 23
  11. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  12. 2024Quantum Many-Body Physics Calculations with Large Language Modelscited by 17
31 more
  1. 2023Supersonic: Learning to Generate Source Code Optimizations in C/C++cited by 16
  2. 2023Can Large Language Models Find And Fix Vulnerable Software?cited by 15
  3. 2025MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimizationcited by 15
  4. 2025Autonomous Code Evolution Meets NP-Completenesscited by 15
  5. 2023Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Modelscited by 11
  6. 2024What is it for a Machine Learning Model to Have a Capability?cited by 10
  7. 2025MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaborationcited by 10
  8. 2024LLMs for Coding and Robotics Educationcited by 9
  9. 2023How should the advent of large language models affect the practice of science?cited by 8
  10. 2024DCA-Bench: A Benchmark for Dataset Curation Agentscited by 6
  11. 2023GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Sciencecited by 5
  12. 2025From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflowcited by 5
  13. 2025MAPS: Multi-Agent Personality Shaping for Collaborative Reasoningcited by 4
  14. 2024Towards Automated Data Sciences with Natural Language and SageCopilot: Practices and Lessons Learnedcited by 3
  15. 2024Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarkscited by 3
  16. 2025On Simulation-Guided LLM-based Code Generation for Safe Autonomous Driving Softwarecited by 3
  17. 2023Evaluating GPT's Programming Capability through CodeWars' Katascited by 2
  18. 2024ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMscited by 2
  19. 2024CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkitcited by 2
  20. 2024SketchFill: Sketch-Guided Code Generation for Imputing Derived Missing Valuescited by 2
  21. 2025Worst-Case Symbolic Constraints Analysis and Generalisation with Large Language Modelscited by 1
  22. 2023Use of probabilistic phrases in a coordination game: human versus GPT-4cited by 0
  23. 2023AtomXR: Streamlined XR Prototyping with Natural Language and Immersive Physical Interactioncited by 0
  24. 2024Inferring Pluggable Types with Machine Learningcited by 0
  25. 2024The why, what, and how of AI-based coding in scientific researchcited by 0
  26. 2025A Showdown of ChatGPT vs DeepSeek in Solving Programming Taskscited by 0
  27. 2025The Impact of AI on Educational Assessment: A Framework for Constructive Alignmentcited by 0
  28. 2026PELLI: Framework to effectively integrate LLMs for quality software generationcited by 0
  29. 2026Code Fingerprints: Disentangled Attribution of LLM-Generated Codecited by 0
  30. 2026Do AI Agents Really Improve Code Readability?cited by 0
  31. 2026Is Solving Better Than Evaluating GenAI Solutions?cited by 0