# The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study

arXiv: https://arxiv.org/abs/2507.03156 · HTML: https://mlsanity.com/paper/2507.03156/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 109 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 30 of 45)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [DebugBench: Evaluating Debugging Capability of Large Language Models](https://mlsanity.com/paper/2401.04621/) (2024)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [On the Effectiveness of Large Language Models in Domain-Specific Code Generation](https://mlsanity.com/paper/2312.01639/) (2023)
- [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021)
- [Navigating the Complexity of Generative AI Adoption in Software Engineering](https://mlsanity.com/paper/2307.06081/) (2023)
- [Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice](https://mlsanity.com/paper/2404.14901/) (2024)
- [A Comparative Analysis of Large Language Models for Code Documentation Generation](https://mlsanity.com/paper/2312.10349/) (2023)
- [AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation](https://mlsanity.com/paper/2305.12050/) (2023)
- [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023)
- [Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions](https://mlsanity.com/paper/2402.11364/) (2024)
- [Better Together? An Evaluation of AI-Supported Code Translation](https://mlsanity.com/paper/2202.07682/) (2022)
- [A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspective](https://mlsanity.com/paper/2007.11354/) (2020)
- [The Role of Generative AI in Software Development Productivity: A Pilot Case Study](https://mlsanity.com/paper/2406.00560/) (2024)
- [Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems](https://mlsanity.com/paper/2209.05948/) (2022)
- [Large Language Models Based Automatic Synthesis of Software Specifications](https://mlsanity.com/paper/2304.09181/) (2023)
- [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024)
- [Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration](https://mlsanity.com/paper/2402.18498/) (2024)
- [A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages](https://mlsanity.com/paper/2308.04477/) (2023)
- [AI-Tutoring in Software Engineering Education](https://mlsanity.com/paper/2404.02548/) (2024)
- [STEAM: Simulating the InTeractive BEhavior of ProgrAMmers for Automatic Bug Fixing](https://mlsanity.com/paper/2308.14460/) (2023)

## Built on by (most cited first, 4 of 4)

- [The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development](https://mlsanity.com/paper/2605.01160/) (2026)
- [Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI](https://mlsanity.com/paper/2607.01418/) (2026)
- [Utilizing LLMs for Industrial Process Automation: A Case Study on Modifying RAPID Programs](https://mlsanity.com/paper/2511.11125/) (2025)
- [Does AI Save Time on Product Design? A Randomized Controlled Experiment of AI Prompt-to-Design Workflows](https://mlsanity.com/paper/2609.26725/) (2026)

## Abstract (arXiv)

Large language model assistants (LLM-assistants) present new opportunities to transform software development. Developers are increasingly adopting these tools across tasks, including coding, testing, debugging, documentation, and design. Yet, despite growing interest, there is no synthesis of how LLM-assistants affect software developer productivity. In this paper, we present a systematic review and mapping of 39 peer-reviewed studies published between January 2014 and December 2024 that examine this impact. Our analysis reveals that the majority of studies report considerable benefits from LLM-assistants, though a notable subset identifies critical risks. Commonly reported gains include accelerated development, minimized code search, and the automation of trivial and repetitive tasks. However, studies also highlight concerns around cognitive offloading and reduced team collaboration. Our study reveals that whether LLM-based assistants improve or degrade code quality remains unresolved, as existing studies report contradictory outcomes contingent on context and evaluation criteria. While the majority of studies (90%) adopt a multi-dimensional perspective by examining at least two SPACE dimensions, reflecting increased awareness of the complexity of developer productivity, only 15% extend beyond three dimensions, indicating substantial room for more integrated evaluations. Satisfaction, Performance, and Efficiency are the most frequently investigated dimensions, whereas Communication and Activity remain underexplored. Most studies are exploratory (59%) and methodologically diverse, but lack longitudinal and team-based evaluations. This review surfaces key research gaps and provides recommendations for future research and practice. All artifacts associated with this study are publicly available at https://zenodo.org/records/18489222