# Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward

arXiv: https://arxiv.org/abs/2406.07765 · HTML: https://mlsanity.com/paper/2406.07765/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023)
- [How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study](https://mlsanity.com/paper/2309.10108/) (2023)
- [Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat](https://mlsanity.com/paper/2401.17163/) (2024)
- [Trust in Generative AI among students: An Exploratory Study](https://mlsanity.com/paper/2310.04631/) (2023)
- [CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming](https://mlsanity.com/paper/2310.09235/) (2023)
- [Better Together? An Evaluation of AI-Supported Code Translation](https://mlsanity.com/paper/2202.07682/) (2022)
- [Practitioners' Expectations on Code Completion](https://mlsanity.com/paper/2301.03846/) (2023)
- [DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2402.13291/) (2024)

## Built on by (most cited first, 30 of 45)

- [Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap](https://mlsanity.com/paper/2410.06107/) (2024)
- [Agentic Refactoring: An Empirical Study of AI Coding Agents](https://mlsanity.com/paper/2511.04824/) (2025)
- [Exploring the Potential of Diffusion Large Language Models in Code Generation](https://mlsanity.com/paper/2509.11252/) (2025)
- [Vibe Coding as a Reconfiguration of Intent Mediation in Software Development: Definition, Implications, and Research Agenda](https://mlsanity.com/paper/2507.21928/) (2025)
- [Harness Engineering for Agentic AI Coding Tools: An Exploratory Study](https://mlsanity.com/paper/2602.14690/) (2026)
- [A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?](https://mlsanity.com/paper/2411.01414/) (2024)
- [TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling](https://mlsanity.com/paper/2510.02758/) (2025)
- [Generating Energy-Efficient Code via Large-Language Models -- Where are we now?](https://mlsanity.com/paper/2509.10099/) (2025)
- [Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code](https://mlsanity.com/paper/2504.13656/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [Exploring the Challenges and Opportunities of AI-assisted Codebase Generation](https://mlsanity.com/paper/2508.07966/) (2025)
- ["My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants](https://mlsanity.com/paper/2508.12285/) (2025)
- [What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects](https://mlsanity.com/paper/2509.07763/) (2025)
- [Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaboration](https://mlsanity.com/paper/2509.10652/) (2025)
- [Can Large Language Models Capture Video Game Engagement?](https://mlsanity.com/paper/2502.04379/) (2025)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [Context Branching for LLM Conversations: A Version Control Approach to Exploratory Programming](https://mlsanity.com/paper/2512.13914/) (2025)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [Accountable Agents in Software Engineering: An Analysis of Terms of Service and a Research Roadmap](https://mlsanity.com/paper/2605.04532/) (2026)
- [Developer Perspectives on Licensing and Copyright Issues Arising from Generative AI for Software Development](https://mlsanity.com/paper/2411.10877/) (2024)
- [From Code to Compliance: Assessing ChatGPT's Utility in Designing an Accessible Webpage -- A Case Study](https://mlsanity.com/paper/2501.03572/) (2025)
- [The State of Generative AI in Software Development: Insights from Literature and a Developer Survey](https://mlsanity.com/paper/2603.16975/) (2026)
- [An Exploratory Study on LLM-Generated Code and Comments in Code Repositories](https://mlsanity.com/paper/2607.01867/) (2026)
- [Multi-task Code LLMs: Data Mix or Model Merge?](https://mlsanity.com/paper/2601.21115/) (2026)
- [Interrogating Design Homogenization in Web Vibe Coding](https://mlsanity.com/paper/2603.13036/) (2026)
- [Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks](https://mlsanity.com/paper/2604.05100/) (2026)
- [Application Modernization with LLMs: Addressing Core Challenges in Reliability, Security, and Quality](https://mlsanity.com/paper/2506.10984/) (2025)
- [Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks](https://mlsanity.com/paper/2508.04125/) (2025)
- [APIDA-Chat: Structured Synthesis of API Search Dialogues to Bootstrap Conversational Agents](https://mlsanity.com/paper/2510.03743/) (2025)
- [Learning to Code with Context: A Study-Based Approach](https://mlsanity.com/paper/2512.05242/) (2025)

## Abstract (arXiv)

Context. The last several years saw the emergence of AI assistants for code - multi-purpose AI-based helpers in software engineering. As they become omnipresent in all aspects of software development, it becomes critical to understand their usage patterns. Objective. We aim to better understand how specifically developers are using AI assistants, why they are not using them in certain parts of their development workflow, and what needs to be improved in the future. Methods. In this work, we carried out a large-scale survey aimed at how AI assistants are used, focusing on specific software development activities and stages. We collected opinions of 481 programmers on five broad activities: (a) implementing new features, (b) writing tests, (c) bug triaging, (d) refactoring, and (e) writing natural-language artifacts, as well as their individual stages. Results. Our results provide a novel comparison of different stages where AI assistants are used that is both comprehensive and detailed. It highlights specific activities that developers find less enjoyable and want to delegate to an AI assistant, e.g., writing tests and natural-language artifacts. We also determine more granular stages where AI assistants are used, such as generating tests and generating docstrings, as well as less studied parts of the workflow, such as generating test data. Among the reasons for not using assistants, there are general aspects like trust and company policies, as well as more concrete issues like the lack of project-size context, which can be the focus of the future research. Conclusion. The provided analysis highlights stages of software development that developers want to delegate and that are already popular for using AI assistants, which can be a good focus for features aimed to help developers right now. The main reasons for not using AI assistants can serve as a guideline for future work.