# Language Models for Code Completion: A Practical Evaluation

arXiv: https://arxiv.org/abs/2402.16197 · HTML: https://mlsanity.com/paper/2402.16197/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 25 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [CM3: A Causal Masked Multimodal Model of the Internet](https://mlsanity.com/paper/2201.07520/) (2022)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries](https://mlsanity.com/paper/2301.01701/) (2023)
- [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021)
- [An Empirical Study on the Usage of Transformer Models for Code Completion](https://mlsanity.com/paper/2108.01585/) (2021)
- [Topic Recommendation for Software Repositories using Multi-label Classification Algorithms](https://mlsanity.com/paper/2010.09116/) (2020)
- [Learning Autocompletion from Real-World Datasets](https://mlsanity.com/paper/2011.04542/) (2020)
- [All You Need Is Logs: Improving Code Completion by Learning from Anonymous IDE Usage Logs](https://mlsanity.com/paper/2205.10692/) (2022)
- [Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study](https://mlsanity.com/paper/2304.12269/) (2023)
- [On the Evaluation of NLP-based Models for Software Engineering](https://mlsanity.com/paper/2203.17166/) (2022)
- [On the Impact of Language Selection for Training and Evaluating Programming Language Models](https://mlsanity.com/paper/2308.13354/) (2023)

## Built on by (most cited first, 30 of 39)

- [A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages](https://mlsanity.com/paper/2410.03981/) (2024)
- [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026)
- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding](https://mlsanity.com/paper/2506.11309/) (2025)
- [Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents](https://mlsanity.com/paper/2411.03455/) (2024)
- [Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study](https://mlsanity.com/paper/2403.15185/) (2024)
- [Planning-Driven Programming: A Large Language Model Programming Workflow](https://mlsanity.com/paper/2411.14503/) (2024)
- [AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion](https://mlsanity.com/paper/2601.19697/) (2026)
- [Scaling Agentic Verifier for Competitive Coding](https://mlsanity.com/paper/2602.04254/) (2026)
- [Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks](https://mlsanity.com/paper/2504.01850/) (2025)
- [Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity](https://mlsanity.com/paper/2601.18345/) (2026)
- [A Transformer-Based Approach for Smart Invocation of Automatic Code Completion](https://mlsanity.com/paper/2405.14753/) (2024)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat](https://mlsanity.com/paper/2507.18515/) (2025)
- [Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time](https://mlsanity.com/paper/2604.00917/) (2026)
- [Detection of LLM-Paraphrased Code and Identification of the Responsible LLM Using Coding Style Features](https://mlsanity.com/paper/2502.17749/) (2025)
- [Beware What You Autocomplete: Forensic Attribution of Backdoored Code Completions](https://mlsanity.com/paper/2607.08011/) (2026)
- [How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning](https://mlsanity.com/paper/2501.17501/) (2025)
- [Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History](https://mlsanity.com/paper/2508.10074/) (2025)
- [The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion](https://mlsanity.com/paper/2508.16131/) (2025)
- [SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion](https://mlsanity.com/paper/2510.17925/) (2025)
- [Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges](https://mlsanity.com/paper/2604.19354/) (2026)
- [Rethinking IDE Customization for Enhanced HAX: A Hyperdimensional Perspective](https://mlsanity.com/paper/2501.02491/) (2025)
- [A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics](https://mlsanity.com/paper/2505.15469/) (2025)
- [Evaluating Language Models on Cross-Language Code Functional Equivalence](https://mlsanity.com/paper/2608.23961/) (2026)
- [Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering](https://mlsanity.com/paper/2509.17096/) (2025)
- [Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs](https://mlsanity.com/paper/2601.10496/) (2026)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion](https://mlsanity.com/paper/2505.13073/) (2025)
- [TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs](https://mlsanity.com/paper/2508.02455/) (2025)

## Abstract (arXiv)

Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions. These models were then evaluated using six standard metrics across twelve programming languages. Next, we conducted a qualitative study of 1690 real-world completion requests to identify the reasons behind the poor model performance. A comparative analysis of the models' performance in online and offline settings was also performed, using benchmark synthetic datasets and two masking strategies. Our findings suggest that while developers utilize code completion across various languages, the best results are achieved for mainstream languages such as Python and Java. InCoder outperformed the other models across all programming languages, highlighting the significance of training data and objectives. Our study also revealed that offline evaluations do not accurately reflect real-world scenarios. Upon qualitative analysis of the model's predictions, we found that 66.3% of failures were due to the models' limitations, 24.4% occurred due to inappropriate model usage in a development context, and 9.3% were valid requests that developers overwrote. Given these findings, we propose several strategies to overcome the current limitations. These include refining training objectives, improving resilience to typographical errors, adopting hybrid approaches, and enhancing implementations and usability.