# Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study arXiv: https://arxiv.org/abs/2304.12269 ยท HTML: https://mlsanity.com/paper/2304.12269/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 51 references that are in the map. - First posted on arXiv 24 Apr 2023. ## Builds on (most cited first, 30 of 32) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019) - [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020) - [CM3: A Causal Masked Multimodal Model of the Internet](https://mlsanity.com/paper/2201.07520/) (2022) - [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018) - [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019) - [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022) - [LambdaNet: Probabilistic Type Inference using Graph Neural Networks](https://mlsanity.com/paper/2005.02161/) (2020) - [Cloze-driven Pretraining of Self-attention Networks](https://mlsanity.com/paper/1903.07785/) (2019) - [Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries](https://mlsanity.com/paper/2301.01701/) (2023) - [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021) - [An Empirical Study on the Usage of Transformer Models for Code Completion](https://mlsanity.com/paper/2108.01585/) (2021) - [Topic Recommendation for Software Repositories using Multi-label Classification Algorithms](https://mlsanity.com/paper/2010.09116/) (2020) - [Empirical Study of Transformers for Source Code](https://mlsanity.com/paper/2010.07987/) (2020) - [Predicting the Objective and Priority of Issue Reports in Software Repositories](https://mlsanity.com/paper/2012.10951/) (2020) - [Combining Program Analysis and Statistical Language Model for Code Statement Completion](https://mlsanity.com/paper/1911.07781/) (2019) - [An Empirical Study on Code Comment Completion](https://mlsanity.com/paper/2107.10544/) (2021) - [On the Evaluation of NLP-based Models for Software Engineering](https://mlsanity.com/paper/2203.17166/) (2022) - [CatIss: An Intelligent Tool for Categorizing Issues Reports using Transformers](https://mlsanity.com/paper/2203.17196/) (2022) ## Built on by (most cited first, 8 of 8) - [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023) - [Language Models for Code Completion: A Practical Evaluation](https://mlsanity.com/paper/2402.16197/) (2024) - [Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study](https://mlsanity.com/paper/2403.15185/) (2024) - [GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models](https://mlsanity.com/paper/2411.05830/) (2024) - [On the Impact of Language Selection for Training and Evaluating Programming Language Models](https://mlsanity.com/paper/2308.13354/) (2023) - [How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning](https://mlsanity.com/paper/2501.17501/) (2025) - [Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion](https://mlsanity.com/paper/2505.13073/) (2025) - [On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study](https://mlsanity.com/paper/2601.23059/) (2026) ## Abstract (arXiv) Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a developer's toolkit. While many have striven to improve the code-understanding abilities of such models, the opposite -- making the code easier to understand -- has not been properly investigated. In this study, we aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. We consider type annotations and comments as two common forms of additional contextual information that often help developers understand code better. For the experiments, we study code completion in two granularity levels; token and line completion and take three recent and large-scale language models for source code: UniXcoder, CodeGPT, and InCoder with five evaluation metrics. Finally, we perform the Wilcoxon Signed Rank test to gauge significance and measure the effect size. Contrary to our expectations, all models perform better if type annotations are removed (albeit the effect sizes are small). For comments, we find that the models perform better in the presence of multi-line comments (again with small effect sizes). Based on our observations, we recommend making proper design choices when training, fine-tuning, or simply selecting such models given the intended data and application. Better evaluations and multi-modal techniques can also be further investigated to improve the practicality and accuracy of auto-completions.