# A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning

arXiv: https://arxiv.org/abs/1909.06983 · HTML: https://mlsanity.com/paper/1909.06983/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 18 of 18)

- [Leveraging Automated Unit Tests for Unsupervised Code Translation](https://mlsanity.com/paper/2110.06773/) (2021)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [DOBF: A Deobfuscation Pre-Training Objective for Programming Languages](https://mlsanity.com/paper/2102.07492/) (2021)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022)
- [Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs](https://mlsanity.com/paper/2103.09499/) (2021)
- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Cross-Language Binary-Source Code Matching with Intermediate Representations](https://mlsanity.com/paper/2201.07420/) (2022)
- [Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach](https://mlsanity.com/paper/2404.14296/) (2024)
- [Syntax-Aware On-the-Fly Code Completion](https://mlsanity.com/paper/2211.04673/) (2022)
- [MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation Learning](https://mlsanity.com/paper/2206.06460/) (2022)
- [Automated Code-centric Software Vulnerability Assessment: How Far Are We? An Empirical Study in C/C++](https://mlsanity.com/paper/2407.17053/) (2024)
- [Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models](https://mlsanity.com/paper/2106.13928/) (2021)
- [A Survey of Deep Learning Models for Structural Code Understanding](https://mlsanity.com/paper/2205.01293/) (2022)
- [DeepAnalyze: Learning to Localize Crashes at Scale](https://mlsanity.com/paper/2109.14326/) (2021)
- [CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Models](https://mlsanity.com/paper/2503.23231/) (2025)
- [Proactive Agentic Whiteboards: Enhancing Diagrammatic Learning](https://mlsanity.com/paper/2512.01234/) (2025)

## Abstract (arXiv)

Code completion, one of the most useful features in the Integrated Development Environments (IDEs), can accelerate software development by suggesting the libraries, APIs, and method names in real-time. Recent studies have shown that statistical language models can improve the performance of code completion tools through learning from large-scale software repositories. However, these models suffer from three major drawbacks: a) The hierarchical structural information of the programs is not fully utilized in the program's representation; b) In programs, the semantic relationships can be very long. Existing recurrent neural networks based language models are not sufficient to model the long-term dependency. c) Existing approaches perform a specific task in one model, which leads to the underuse of the information from related tasks. To address these challenges, in this paper, we propose a self-attentional neural architecture for code completion with multi-task learning. To utilize the hierarchical structural information of the programs, we present a novel method that considers the path from the predicting node to the root node. To capture the long-term dependency in the input programs, we adopt a self-attentional architecture based network as the base language model. To enable the knowledge sharing between related tasks, we creatively propose a Multi-Task Learning (MTL) framework to learn two related tasks in code completion jointly. Experiments on three real-world datasets demonstrate the effectiveness of our model when compared with state-of-the-art methods.