mlsanityData to 14 Oct 2026

The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study

arXiv 2507.03156 ↗First posted 3 Jul 2025cs.SE cs.AI cs.HCcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language model assistants (LLM-assistants) present new opportunities to transform software development. Developers are increasingly adopting these tools across tasks, including coding, testing, debugging, documentation, and design. Yet, despite growing interest, there is no synthesis of how LLM-assistants affect software developer productivity. In this paper, we present a systematic review and mapping of 39 peer-reviewed studies published between January 2014 and December 2024 that examine this impact. Our analysis reveals that the majority of studies report considerable benefits from LLM-assistants, though a notable subset identifies critical risks. Commonly reported gains include accelerated development, minimized code search, and the automation of trivial and repetitive tasks. However, studies also highlight concerns around cognitive offloading and reduced team collaboration. Our study reveals that whether LLM-based assistants improve or degrade code quality remains unresolved, as existing studies report contradictory outcomes contingent on context and evaluation criteria. While the majority of studies (90%) adopt a multi-dimensional perspective by examining at least two SPACE dimensions, reflecting increased awareness of the complexity of developer productivity, only 15% extend beyond three dimensions, indicating substantial room for more integrated evaluations. Satisfaction, Performance, and Efficiency are the most frequently investigated dimensions, whereas Communication and Activity remain underexplored. Most studies are exploratory (59%) and methodologically diverse, but lack longitudinal and team-based evaluations. This review surfaces key research gaps and provides recommendations for future research and practice. All artifacts associated with this study are publicly available at https://zenodo.org/records/18489222

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 45 of its 109 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

45 of 109 references.

References · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  3. 2023Using an LLM to Help With Code Understandingcited by 228
  4. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  5. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
  6. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 166
  7. 2022Productivity Assessment of Neural Code Completioncited by 98
  8. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  9. 2024DebugBench: Evaluating Debugging Capability of Large Language Modelscited by 76
  10. 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
  11. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  12. 2023ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Designcited by 66
33 more
  1. 2023On the Effectiveness of Large Language Models in Domain-Specific Code Generationcited by 44
  2. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 42
  3. 2023Navigating the Complexity of Generative AI Adoption in Software Engineeringcited by 32
  4. 2024Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practicecited by 25
  5. 2023A Comparative Analysis of Large Language Models for Code Documentation Generationcited by 24
  6. 2023AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluationcited by 21
  7. 2023Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completionscited by 20
  8. 2024Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactionscited by 20
  9. 2022Better Together? An Evaluation of AI-Supported Code Translationcited by 15
  10. 2020A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspectivecited by 14
  11. 2024The Role of Generative AI in Software Development Productivity: A Pilot Case Studycited by 14
  12. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 13
  13. 2023Large Language Models Based Automatic Synthesis of Software Specificationscited by 13
  14. 2024Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Taskscited by 13
  15. 2024Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaborationcited by 13
  16. 2023A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languagescited by 12
  17. 2024AI-Tutoring in Software Engineering Educationcited by 11
  18. 2023STEAM: Simulating the InTeractive BEhavior of ProgrAMmers for Automatic Bug Fixingcited by 9
  19. 2023Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomescited by 9
  20. 2023How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineeringcited by 9
  21. 2024From Today's Code to Tomorrow's Symphony: The AI Transformation of Developer's Routine by 2030cited by 8
  22. 2021Search Strategy Formulation for Systematic Reviews: issues, challenges and opportunitiescited by 6
  23. 2023ChatGPT as a Software Development Bot: A Project-based Studycited by 5
  24. 2024Systematic Literature Review of Commercial Participation in Open Source Softwarecited by 5
  25. 2017What happens when software developers are (un)happycited by 4
  26. 2024LLM-Cure: LLM-based Competitor User Review Analysis for Feature Enhancementcited by 4
  27. 2013Developer Experience: Concept and Definitioncited by 3
  28. 2020Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivitycited by 3
  29. 2021How Developers and Managers Define and Trade Productivity for Qualitycited by 3
  30. 2024The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistantscited by 3
  31. 2018A Systematic Review of Productivity Factors in Software Developmentcited by 2
  32. 2021Improving Productivity through Corporate Hackathons: A Multiple Case Study of Two Large-scale Agile Organizationscited by 1
  33. 2023Impostor Phenomenon in Software Engineerscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.