# Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming arXiv: https://arxiv.org/abs/2210.14306 ยท HTML: https://mlsanity.com/paper/2210.14306/ ## Key facts - Cited by 91 papers in the map (data through 14 Oct 2026). - Builds on 14 of its 34 references that are in the map. - First posted on arXiv 25 Oct 2022. ## Builds on (most cited first, 14 of 14) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021) - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021) - [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022) - [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022) - [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022) - [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022) - [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022) - [Perfection Not Required? Human-AI Partnerships in Code Translation](https://mlsanity.com/paper/2104.03820/) (2021) - [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022) ## Built on by (most cited first, 30 of 91) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024) - [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023) - [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023) - [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023) - [XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models](https://mlsanity.com/paper/2411.15100/) (2024) - [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024) - [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026) - [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024) - [ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers](https://mlsanity.com/paper/2305.14591/) (2023) - [RepoFusion: Training Code Models to Understand Your Repository](https://mlsanity.com/paper/2306.10998/) (2023) - [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023) - [Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support](https://mlsanity.com/paper/2502.18658/) (2025) - [StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code](https://mlsanity.com/paper/2306.04556/) (2023) - [Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review](https://mlsanity.com/paper/2307.02503/) (2023) - [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026) - [Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts](https://mlsanity.com/paper/2308.03921/) (2023) - [How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging](https://mlsanity.com/paper/2310.05292/) (2023) - [The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers](https://mlsanity.com/paper/2404.02806/) (2024) - [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023) - [Prediction-Powered Ranking of Large Language Models](https://mlsanity.com/paper/2402.17826/) (2024) - [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024) - [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023) - [Scientists' Perspectives on the Potential for Generative AI in their Fields](https://mlsanity.com/paper/2304.01420/) (2023) - [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023) - [Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming](https://mlsanity.com/paper/2306.05153/) (2023) - [Learning Personalized Decision Support Policies](https://mlsanity.com/paper/2304.06701/) (2023) - [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024) - [Evaluating ChatGPT and GPT-4 for Visual Programming](https://mlsanity.com/paper/2308.02522/) (2023) - [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024) ## Abstract (arXiv) Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.