mlsanityData to 7 Oct 2026

Productivity Assessment of Neural Code Completion

arXiv 2205.06537 ↗First posted 13 May 2022cs.SE cs.CL cs.HCcited by 97

Not on the map. Why?

Abstract (arXiv)

Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers' productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

13 of 20 references.

References · 13

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  6. 2019SPoC: Search-based Pseudocode to Codecited by 176
  7. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  8. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42
  9. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 41
  10. 2020Fast and Memory-Efficient Neural Code Completioncited by 38
  11. 2020Learning Autocompletion from Real-World Datasetscited by 19
  12. 2021Improving Code Autocompletion with Transfer Learningcited by 6
1 more
  1. 2020Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 21; 2024: 23; 2025: 17; 2026: 29.

97 papers.

Sort:

By area:

  1. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  2. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  3. 2023Using an LLM to Help With Code Understandingcited by 222
  4. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  5. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  6. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  7. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  8. 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
  9. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  10. 2023Using Large Language Models to Generate JUnit Tests: An Empirical Studycited by 53
  11. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  12. 2024Calibration and Correctness of Language Models for Codecited by 49
38 more
  1. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  2. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  3. 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 44
  4. 2022What is it like to program with artificial intelligence?cited by 39
  5. 2024Language Models for Code Completion: A Practical Evaluationcited by 38
  6. 2023SALLM: Security Assessment of Generated Codecited by 35
  7. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  8. 2026Agentic Much? Adoption of Coding Agents on GitHubcited by 31
  9. 2023Large Language Models of Code Fail at Completing Code with Potential Bugscited by 28
  10. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  11. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  12. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  13. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  14. 2023Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completionscited by 20
  15. 2023AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluationcited by 20
  16. 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
  17. 2023Exploring and Characterizing Large Language Models For Embedded System Development and Debuggingcited by 20
  18. 2024Prompts Are Programs Too! Understanding How Developers Build Software Containing Promptscited by 19
  19. 2024Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projectscited by 18
  20. 2025EvoC2Rust: A Skeleton-guided Framework for Project-Level C-to-Rust Translationcited by 18
  21. 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 17
  22. 2026AIDev: Studying AI Coding Agents on GitHubcited by 15
  23. 2022Aligning Offline Metrics and Human Judgments of Value for Code Generation Modelscited by 14
  24. 2023A Static Evaluation of Code Completion by Large Language Modelscited by 14
  25. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  26. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  27. 2024Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Taskscited by 13
  28. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 12
  29. 2024Studying LLM Performance on Closed- and Open-source Datacited by 11
  30. 2023CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination Using Large Generative Modelscited by 10
  31. 2024When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inferencecited by 10
  32. 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Developmentcited by 10
  33. 2024The Fault in our Stars: Quality Assessment of Code Generation Benchmarkscited by 9
  34. 2026AgentStepper: Interactive Debugging of Software Development Agentscited by 8
  35. 2025The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impairedcited by 7
  36. 2024ChatGPT Code Detection: Techniques for Uncovering the Source of Codecited by 6
  37. 2024EmbedGenius: Towards Automated Software Development for Generic Embedded IoT Systemscited by 6
  38. 2026Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activitycited by 6

50 most cited of 97 All 97 (JSON)