Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
Not on the map. Why?
Abstract (arXiv)
The integration of Artificial Intelligence (AI) into Integrated Development Environments (IDEs) is reshaping software development, fundamentally altering how developers interact with their tools. This shift marks the emergence of Human-AI Experience in Integrated Development Environment (in-IDE HAX), a field that explores the evolving dynamics of Human-Computer Interaction in AI-assisted coding environments. Despite rapid adoption, research on in-IDE HAX remains fragmented, which highlights the need for a unified overview of current practices, challenges, and opportunities. To provide a structured overview of existing research, we conduct a systematic literature review of 90 studies, summarizing current findings and outlining areas for further investigation. We organize key insights from reviewed studies into three aspects: Impact, Design, and Quality of AI-based systems inside IDEs. Impact findings show that AI-assisted coding enhances developer productivity but also introduces challenges, such as verification overhead and over-reliance. Design studies show that effective interfaces surface context, provide explanations and transparency of suggestion, and support user control. Quality studies document risks in correctness, maintainability, and security. For future research, priorities include productivity studies, design of assistance, and audit of AI-generated code. The agenda calls for larger and longer evaluations, stronger audit and verification assets, broader coverage across the software life cycle, and adaptive assistance under user control.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 68 of its 127 references that are in the map. First posted on arXiv 8 Mar 2025.
Builds on
68 of 127 references.
References · 68
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
- 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
- 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
- 2022Productivity Assessment of Neural Code Completioncited by 98
- 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 96
- 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
- 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
- 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
- 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
- 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
38 more
- 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
- 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 67
- 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
- 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
- 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 56
- 2024Calibration and Correctness of Language Models for Codecited by 49
- 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 48
- 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
- 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 44
- 2024Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Aheadcited by 42
- 2024Language Models for Code Completion: A Practical Evaluationcited by 39
- 2023Investigating and Designing for Trust in AI-powered Code Generation Toolscited by 34
- 2023Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Promptscited by 30
- 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 30
- 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
- 2023On the Design of AI-powered Code Assistants for Notebookscited by 28
- 2024An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Projectcited by 28
- 2024Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concernscited by 28
- 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
- 2023Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chainscited by 24
- 2023How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Studycited by 24
- 2023Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completionscited by 20
- 2023Interactive Text-to-SQL Generation via Editable Step-by-Step Explanationscited by 19
- 2024Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projectscited by 18
- 2021Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebookscited by 17
- 2023Trust in Generative AI among students: An Exploratory Studycited by 16
- 2023CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programmingcited by 16
- 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
- 2022Better Together? An Evaluation of AI-Supported Code Translationcited by 15
- 2023A User-centered Security Evaluation of Copilotcited by 15
- 2022How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilotcited by 14
- 2022"It would work for me too": How Online Communities Shape Software Developers' Trust in AI-Powered Code Generation Toolscited by 14
- 2023Practices and Challenges of Using GitHub Copilot: An Empirical Studycited by 14
- 2024Ivie: Lightweight Anchored Explanations of Just-Generated Codecited by 14
- 2024Multi-line AI-assisted Code Authoringcited by 13
- 2024Copilot Evaluation Harness: Evaluating LLM-Guided Software Programmingcited by 13
- 2024Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysiscited by 12
- 2024A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actionscited by 12
50 most cited of 68 All 68 (JSON)
Built on by
Citing papers per year: 2025: 5.
5 papers.
- 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 12
- 2025"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistantscited by 4
- 2025Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Studycited by 1
- 2025Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agentscited by 0
- 2025Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Taskscited by 0
- 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding30 Nov 2025
- 2025"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants17 Aug 2025
- 2025Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks6 Aug 2025
- 2025Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study5 Jul 2025
- 2025Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agents24 Jun 2025