mlsanityData to 7 Oct 2026

Comparing large language models and human programmers for generating programming code

arXiv 2403.00894 ↗First posted 1 Mar 2024cs.SE cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We systematically evaluated the performance of seven large language models in generating programming code using various prompt strategies, programming languages, and task difficulties. GPT-4 substantially outperforms other large language models, including Gemini Ultra and Claude 2. The coding performance of GPT-4 varies considerably with different prompt strategies. In most LeetCode and GeeksforGeeks coding contests evaluated in this study, GPT-4 employing the optimal prompt strategy outperforms 85 percent of human participants. Additionally, GPT-4 demonstrates strong capabilities in translating code between different programming languages and in learning from past errors. The computational efficiency of the code generated by GPT-4 is comparable to that of human programmers. These results suggest that GPT-4 has the potential to serve as a reliable assistant in programming code generation and software development.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

16 of 32 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  10. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  11. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  12. 2023Is GPT-4 a Good Data Analyst?cited by 69
4 more
  1. 2023Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutorscited by 38
  2. 2023ChatGPT Participates in a Computer Science Examcited by 19
  3. 2023Evaluating ChatGPT and GPT-4 for Visual Programmingcited by 13
  4. 2023The potential of LLMs for coding with low-resource and domain-specific programming languagescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 1.

9 papers.

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  3. 2024GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasetscited by 12
  4. 2025The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budgetcited by 11
  5. 2026LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loopscited by 8
  6. 2025Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"cited by 5
  7. 2025CorrectHDL: Agentic HDL Design with LLMs Leveraging High-Level Synthesis as Referencecited by 3
  8. 2025Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?cited by 1
  9. 2024Inferring Pluggable Types with Machine Learningcited by 0