mlsanityData to 7 Oct 2026

Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming

arXiv 2210.14306 ↗First posted 25 Oct 2022cs.SE cs.HC cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

14 of 34 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  8. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  9. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  10. 2022Productivity Assessment of Neural Code Completioncited by 97
  11. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  12. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
2 more
  1. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42
  2. 2022What is it like to program with artificial intelligence?cited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 16; 2025: 24; 2026: 28.

91 papers.

Sort:

By area:

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  4. 2023Using an LLM to Help With Code Understandingcited by 222
  5. 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
  6. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  7. 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 46
  8. 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 44
  9. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  10. 2023RepoFusion: Training Code Models to Understand Your Repositorycited by 40
  11. 2026Code as Agent Harnesscited by 40
  12. 2023Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutorscited by 38
38 more
  1. 2025Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Supportcited by 33
  2. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  3. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  4. 2026Agentic Much? Adoption of Coding Agents on GitHubcited by 31
  5. 2023Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Promptscited by 30
  6. 2023Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validationcited by 29
  7. 2023How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debuggingcited by 29
  8. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  9. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  10. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  11. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  12. 2023Scientists' Perspectives on the Potential for Generative AI in their Fieldscited by 22
  13. 2023Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completionscited by 20
  14. 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
  15. 2023Learning Personalized Decision Support Policiescited by 13
  16. 2023Evaluating ChatGPT and GPT-4 for Visual Programmingcited by 13
  17. 2024Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Taskscited by 13
  18. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 13
  19. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  20. 2023Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Modelscited by 11
  21. 2024AI on My Shoulder: Supporting Emotional Labor in Front-Office Roles with an LLM-based Empathetic Coworkercited by 11
  22. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 11
  23. 2023R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intentscited by 10
  24. 2025ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulationcited by 10
  25. 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Developmentcited by 10
  26. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  27. 2023Seeing Seeds Beyond Weeds: Green Teaming Generative AI for Beneficial Usescited by 8
  28. 2023Can GPT-4 Replicate Empirical Software Engineering Research?cited by 7
  29. 2024Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketchescited by 7
  30. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 7
  31. 2025The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impairedcited by 7
  32. 2026Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activitycited by 6
  33. 2024A Transformer-Based Approach for Smart Invocation of Automatic Code Completioncited by 5
  34. 2025TableTalk: Scaffolding Spreadsheet Development with a Language Agentcited by 5
  35. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  36. 2025Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Softwarecited by 5
  37. 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generationcited by 5
  38. 2025The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Studycited by 4

50 most cited of 91 All 91 (JSON)