# Evaluation of the Programming Skills of Large Language Models

arXiv: https://arxiv.org/abs/2405.14388 · HTML: https://mlsanity.com/paper/2405.14388/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 8 of 8)

- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023)
- [A Serious Game for Simulating Cyberattacks to Teach Cybersecurity](https://mlsanity.com/paper/2305.03062/) (2023)
- [CyMed: A Framework for Testing Cybersecurity of Connected Medical Devices](https://mlsanity.com/paper/2310.03583/) (2023)
- [Divide, Conquer and Verify: Improving Symbolic Execution Performance](https://mlsanity.com/paper/2310.03598/) (2023)

## Built on by (most cited first, 2 of 2)

- [Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations](https://mlsanity.com/paper/2408.05128/) (2024)
- [How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow](https://mlsanity.com/paper/2507.10818/) (2025)

## Abstract (arXiv)

The advent of Large Language Models (LLM) has revolutionized the efficiency and speed with which tasks are completed, marking a significant leap in productivity through technological innovation. As these chatbots tackle increasingly complex tasks, the challenge of assessing the quality of their outputs has become paramount. This paper critically examines the output quality of two leading LLMs, OpenAI's ChatGPT and Google's Gemini AI, by comparing the quality of programming code generated in both their free versions. Through the lens of a real-world example coupled with a systematic dataset, we investigate the code quality produced by these LLMs. Given their notable proficiency in code generation, this aspect of chatbot capability presents a particularly compelling area for analysis. Furthermore, the complexity of programming code often escalates to levels where its verification becomes a formidable task, underscoring the importance of our study. This research aims to shed light on the efficacy and reliability of LLMs in generating high-quality programming code, an endeavor that has significant implications for the field of software development and beyond.