# Learning Autocompletion from Real-World Datasets

arXiv: https://arxiv.org/abs/2011.04542 · HTML: https://mlsanity.com/paper/2011.04542/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 20 of 20)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Language Models for Code Completion: A Practical Evaluation](https://mlsanity.com/paper/2402.16197/) (2024)
- [Large Language Models of Code Fail at Completing Code with Potential Bugs](https://mlsanity.com/paper/2306.03438/) (2023)
- [Counterfactual Explanations for Models of Code](https://mlsanity.com/paper/2111.05711/) (2021)
- [All You Need Is Logs: Improving Code Completion by Learning from Anonymous IDE Usage Logs](https://mlsanity.com/paper/2205.10692/) (2022)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code](https://mlsanity.com/paper/2305.04106/) (2023)
- [Improving Code Autocompletion with Transfer Learning](https://mlsanity.com/paper/2105.05991/) (2021)
- [Evaluating Generative Patent Language Models](https://mlsanity.com/paper/2206.14578/) (2022)
- [GPoeT-2: A GPT-2 Based Poem Generator](https://mlsanity.com/paper/2205.08847/) (2022)
- [MPI-rical: Data-Driven MPI Distributed Parallelism Assistance with Transformers](https://mlsanity.com/paper/2305.09438/) (2023)
- [A Critical Study of What Code-LLMs (Do Not) Learn](https://mlsanity.com/paper/2406.11930/) (2024)
- [Neural Machine Translation for Code Generation](https://mlsanity.com/paper/2305.13504/) (2023)
- [Sensitivity Analysis on Transferred Neural Architectures of BERT and GPT-2 for Financial Sentiment Analysis](https://mlsanity.com/paper/2207.03037/) (2022)
- [FDI: Attack Neural Code Generation Systems through User Feedback Channel](https://mlsanity.com/paper/2408.04194/) (2024)
- [Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion](https://mlsanity.com/paper/2505.13073/) (2025)

## Abstract (arXiv)

Code completion is a popular software development tool integrated into all major IDEs. Many neural language models have achieved promising results in completion suggestion prediction on synthetic benchmarks. However, a recent study When Code Completion Fails: a Case Study on Real-World Completions demonstrates that these results may not translate to improvements in real-world performance. To combat this effect, we train models on real-world code completion examples and find that these models outperform models trained on committed source code and working version snapshots by 12.8% and 13.8% accuracy respectively. We observe this improvement across modeling technologies and show through A/B testing that it corresponds to a 6.2% increase in programmers' actual autocompletion usage. Furthermore, our study characterizes a large corpus of logged autocompletion usages to investigate why training on real-world examples leads to stronger models.