mlsanityData to 7 Oct 2026

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

arXiv 2303.17125 ↗First posted 30 Mar 2023cs.SE cs.AI cs.HCcited by 75

Not on the map. Why?

Abstract (arXiv)

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 15 of its 64 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

15 of 64 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  3. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  4. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  5. 2022Productivity Assessment of Neural Code Completioncited by 97
  6. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 95
  7. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  8. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 81
  9. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 41
  10. 2023On the Design of AI-powered Code Assistants for Notebookscited by 28
  11. 2022"It would work for me too": How Online Communities Shape Software Developers' Trust in AI-Powered Code Generation Toolscited by 14
  12. 2022Understanding Skills for OSS Communities on GitHubcited by 9
3 more
  1. 2023A Qualitative Study on the Implementation Design Decisions of Developerscited by 7
  2. 2018Towards a Theory of Software Development Expertisecited by 5
  3. 2021Leaving My Fingerprints: Motivations and Challenges of Contributing to OSS for Social Goodcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 36; 2026: 24.

75 papers.

Sort:

By area:

  1. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
  2. 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 44
  3. 2025Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Supportcited by 33
  4. 2026Agentic Much? Adoption of Coding Agents on GitHubcited by 31
  5. 2024The Future of AI-Driven Software Engineeringcited by 23
  6. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  7. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  8. 2025Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Studycited by 16
  9. 2025Exploring the Potential of Diffusion Large Language Models in Code Generationcited by 16
  10. 2024Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Taskscited by 13
  11. 2024RLSF: Fine-tuning LLMs via Symbolic Feedbackcited by 13
  12. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 13
38 more
  1. 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Developmentcited by 10
  2. 2023Can GPT-4 Replicate Empirical Software Engineering Research?cited by 7
  3. 2024Can Developers Prompt? A Controlled Experiment for Code Documentation Generationcited by 7
  4. 2025The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impairedcited by 7
  5. 2025Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?cited by 7
  6. 2024CursorCore: Assist Programming through Aligning Anythingcited by 6
  7. 2024Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Modelscited by 6
  8. 2025Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Taskscited by 6
  9. 2026Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activitycited by 6
  10. 2025TableTalk: Scaffolding Spreadsheet Development with a Language Agentcited by 5
  11. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  12. 2025Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Softwarecited by 5
  13. 2025Generating Energy-Efficient Code via Large-Language Models -- Where are we now?cited by 5
  14. 2025The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Studycited by 4
  15. 2025Exploring the Challenges and Opportunities of AI-assisted Codebase Generationcited by 4
  16. 2025"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistantscited by 4
  17. 2024ChatGPT in Data Visualization Education: A Student Perspectivecited by 3
  18. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 3
  19. 2025Leveraging ASIC AI Chips for Homomorphic Encryptioncited by 3
  20. 2025From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Papercited by 3
  21. 2025On Simulation-Guided LLM-based Code Generation for Safe Autonomous Driving Softwarecited by 3
  22. 2025Identifying, Evaluating, and Mitigating Risks of AI Thought Partnershipscited by 3
  23. 2025Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generationcited by 3
  24. 2025Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaborationcited by 3
  25. 2026Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answeringcited by 3
  26. 2026Humans are Missing from AI Coding Agent Researchcited by 3
  27. 2024Developer Perspectives on Licensing and Copyright Issues Arising from Generative AI for Software Developmentcited by 2
  28. 2025Assessing LLMs for Front-end Software Architecture Knowledgecited by 2
  29. 2025CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Modelscited by 2
  30. 2025Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directionscited by 2
  31. 2025A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metricscited by 2
  32. 2025How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflowcited by 2
  33. 2025Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspectivecited by 2
  34. 2025Coding With AI: From a Reflection on Industrial Practices to Future Computer Science and Software Engineering Educationcited by 2
  35. 2026An Exploratory Study on LLM-Generated Code and Comments in Code Repositoriescited by 2
  36. 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 2
  37. 2024Test Case Features as Hyper-heuristics for Inductive Programmingcited by 1
  38. 2024Retrieval-augmented code completion for local projects using large language modelscited by 1

50 most cited of 75 All 75 (JSON)