# Syntax-Aware On-the-Fly Code Completion

arXiv: https://arxiv.org/abs/2211.04673 · HTML: https://mlsanity.com/paper/2211.04673/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [Generative Code Modeling with Graphs](https://mlsanity.com/paper/1805.08490/) (2018)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Code Completion with Neural Attention and Pointer Networks](https://mlsanity.com/paper/1711.09573/) (2017)
- [Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs](https://mlsanity.com/paper/2103.09499/) (2021)
- [A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning](https://mlsanity.com/paper/1909.06983/) (2019)
- [Towards Full-line Code Completion with Neural Language Models](https://mlsanity.com/paper/2009.08603/) (2020)

## Built on by (most cited first, 10 of 10)

- [Human-In-the-Loop Software Development Agents](https://mlsanity.com/paper/2411.12924/) (2024)
- [Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review](https://mlsanity.com/paper/2307.02503/) (2023)
- [M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation](https://mlsanity.com/paper/2410.21157/) (2024)
- [Constrained Decoding for Fill-in-the-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars](https://mlsanity.com/paper/2402.17988/) (2024)
- [An Empirical Study on Capability of Large Language Models in Understanding Code Semantics](https://mlsanity.com/paper/2407.03611/) (2024)
- [Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension](https://mlsanity.com/paper/2404.08885/) (2024)
- [Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassian](https://mlsanity.com/paper/2501.11264/) (2025)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [Approximately Aligned Decoding](https://mlsanity.com/paper/2410.01103/) (2024)
- [Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion](https://mlsanity.com/paper/2505.13073/) (2025)

## Abstract (arXiv)

Code completion aims to help improve developers' productivity by suggesting the next code tokens from a given context. Various approaches have been proposed to incorporate abstract syntax tree (AST) information for model training, ensuring that code completion is aware of the syntax of the programming languages. However, existing syntax-aware code completion approaches are not on-the-fly, as we found that for every two-thirds of characters that developers type, AST fails to be extracted because it requires the syntactically correct source code, limiting its practicality in real-world scenarios. On the other hand, existing on-the-fly code completion does not consider syntactic information yet. In this paper, we propose PyCoder to leverage token types, a kind of lightweight syntactic information, which is readily available and aligns with the natural order of source code. Our PyCoder is trained in a multi-task training manner so that by learning the supporting task of predicting token types during the training phase, the models achieve better performance on predicting tokens and lines of code without the need for token types in the inference phase. Comprehensive experiments show that PyCoder achieves the first rank on the CodeXGLUE leaderboard with an accuracy of 77.12% for the token-level predictions, which is 0.43%-24.25% more accurate than baselines. In addition, PyCoder achieves an exact match of 43.37% for the line-level predictions, which is 3.63%-84.73% more accurate than baselines. These results lead us to conclude that token type information (an alternative to syntactic information) that is rarely used in the past can greatly improve the performance of code completion approaches, without requiring the syntactically correct source code like AST-based approaches do. Our PyCoder is publicly available on HuggingFace and GitHub.