# "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants

arXiv: https://arxiv.org/abs/2508.12285 · HTML: https://mlsanity.com/paper/2508.12285/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 74 references that are in the map.
- First posted on arXiv 17 Aug 2025.

## Builds on (most cited first, 30 of 38)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Stealthy Backdoor Attack for Code Models](https://mlsanity.com/paper/2301.02496/) (2023)
- [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024)
- [Unveiling Memorization in Code Models](https://mlsanity.com/paper/2308.09932/) (2023)
- [On the Design of AI-powered Code Assistants for Notebooks](https://mlsanity.com/paper/2301.11178/) (2023)
- [Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code](https://mlsanity.com/paper/2403.07506/) (2024)
- [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023)
- [Compressing Pre-trained Models of Code into 3 MB](https://mlsanity.com/paper/2208.07120/) (2022)
- [Greening Large Language Models of Code](https://mlsanity.com/paper/2309.04076/) (2023)
- [Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead](https://mlsanity.com/paper/2404.04566/) (2024)
- [Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems](https://mlsanity.com/paper/2209.05948/) (2022)
- [MORepair: Teaching LLMs to Repair Code via Multi-Objective Fine-tuning](https://mlsanity.com/paper/2404.12636/) (2024)
- [Exploring the Problems, their Causes and Solutions of AI Pair Programming: A Study on GitHub and Stack Overflow](https://mlsanity.com/paper/2311.01020/) (2023)
- [When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference](https://mlsanity.com/paper/2401.09964/) (2024)
- [When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair](https://mlsanity.com/paper/2412.01072/) (2024)
- [Mining User Opinions in Mobile App Reviews: A Keyword-based Approach](https://mlsanity.com/paper/1505.04657/) (2015)
- [CHRONOS: Time-Aware Zero-Shot Identification of Libraries from Vulnerability Reports](https://mlsanity.com/paper/2301.03944/) (2023)
- [Evaluating SZZ Implementations: An Empirical Study on the Linux Kernel](https://mlsanity.com/paper/2308.05060/) (2023)

## Built on by (most cited first, 4 of 4)

- ["Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors](https://mlsanity.com/paper/2509.22040/) (2025)
- [SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios](https://mlsanity.com/paper/2509.22097/) (2025)
- [How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study](https://mlsanity.com/paper/2607.10856/) (2026)
- [Comparing Developer and LLM Biases in Code Evaluation](https://mlsanity.com/paper/2603.24586/) (2026)

## Abstract (arXiv)

This paper aims to explore fundamental questions in the era when AI coding assistants like GitHub Copilot are widely adopted: what do developers truly value and criticize in AI coding assistants, and what does this reveal about their needs and expectations in real-world software development? Unlike previous studies that conduct observational research in controlled and simulated environments, we analyze extensive, first-hand user reviews of AI coding assistants, which capture developers' authentic perspectives and experiences drawn directly from their actual day-to-day work contexts. We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace. Although they only account for 1.64% of all extensions, we observe a surge in these assistants: over 90% of them are released within the past two years. We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants. We manually annotate each review's attitude when mentioning certain aspects of coding assistants, yielding nuanced insights into user satisfaction and dissatisfaction regarding specific features, concerns, and overall tool performance. Built on top of the findings-including how users demand not just intelligent suggestions but also context-aware, customizable, and resource-efficient interactions-we propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs.