# Comparing Software Developers with ChatGPT: An Empirical Investigation

arXiv: https://arxiv.org/abs/2305.11837 · HTML: https://mlsanity.com/paper/2305.11837/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [Fault-Aware Neural Code Rankers](https://mlsanity.com/paper/2206.03865/) (2022)
- [Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming](https://mlsanity.com/paper/2111.07875/) (2021)
- [An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode](https://mlsanity.com/paper/2208.08603/) (2022)
- [Supporting Contextual Conversational Agent-Based Software Development](https://mlsanity.com/paper/2305.00885/) (2023)

## Built on by (most cited first, 11 of 11)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology](https://mlsanity.com/paper/2402.01411/) (2024)
- [Human-Artificial Interaction in the Age of Agentic AI: A System-Theoretical Approach](https://mlsanity.com/paper/2502.14000/) (2025)
- [Autonomous Agents in Software Development: A Vision Paper](https://mlsanity.com/paper/2311.18440/) (2023)
- [Designing explainable artificial intelligence with active inference: A framework for transparent introspection and decision-making](https://mlsanity.com/paper/2306.04025/) (2023)
- [Evaluating Privacy Questions From Stack Overflow: Can ChatGPT Compete?](https://mlsanity.com/paper/2306.11174/) (2023)
- [Assessing the Code Clone Detection Capability of Large Language Models](https://mlsanity.com/paper/2407.02402/) (2024)
- [Learning to Code with Context: A Study-Based Approach](https://mlsanity.com/paper/2512.05242/) (2025)
- [The Expert Validation Framework (EVF): Enabling Domain Expert Control in AI Engineering](https://mlsanity.com/paper/2601.12327/) (2026)
- [PELLI: Framework to effectively integrate LLMs for quality software generation](https://mlsanity.com/paper/2602.10808/) (2026)
- [The Perfection Paradox: From Architect to Curator in AI-Assisted API Design](https://mlsanity.com/paper/2603.12475/) (2026)

## Abstract (arXiv)

The advent of automation in particular Software Engineering (SE) tasks has transitioned from theory to reality. Numerous scholarly articles have documented the successful application of Artificial Intelligence to address issues in areas such as project management, modeling, testing, and development. A recent innovation is the introduction of ChatGPT, an ML-infused chatbot, touted as a resource proficient in generating programming codes and formulating software testing strategies for developers and testers respectively. Although there is speculation that AI-based computation can increase productivity and even substitute software engineers in software development, there is currently a lack of empirical evidence to verify this. Moreover, despite the primary focus on enhancing the accuracy of AI systems, non-functional requirements including energy efficiency, vulnerability, fairness (i.e., human bias), and safety frequently receive insufficient attention. This paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration, enhancing the reliability of AI-based methods, and understanding task suitability for humans or AI. Furthermore, it facilitates the effective implementation of cooperative work structures and human-in-the-loop processes. This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics. The empirical study includes a case of assessing ChatGPT-generated code versus code produced by developers and uploaded in Leetcode.