# Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models

arXiv: https://arxiv.org/abs/2106.13928 · HTML: https://mlsanity.com/paper/2106.13928/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [Generative Code Modeling with Graphs](https://mlsanity.com/paper/1805.08490/) (2018)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [Code Completion with Neural Attention and Pointer Networks](https://mlsanity.com/paper/1711.09573/) (2017)
- [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016)
- [Fast and Memory-Efficient Neural Code Completion](https://mlsanity.com/paper/2004.13651/) (2020)
- [A General Path-Based Representation for Predicting Program Properties](https://mlsanity.com/paper/1803.09544/) (2018)
- [A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning](https://mlsanity.com/paper/1909.06983/) (2019)
- [Sequence Model Design for Code Completion in the Modern IDE](https://mlsanity.com/paper/2004.05249/) (2020)
- [Parameter-Free Probabilistic API Mining across GitHub](https://mlsanity.com/paper/1512.05558/) (2015)

## Built on by (most cited first, 5 of 5)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Prompt-based Code Completion via Multi-Retrieval Augmented Generation](https://mlsanity.com/paper/2405.07530/) (2024)
- [Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware](https://mlsanity.com/paper/2505.05057/) (2025)
- [FDI: Attack Neural Code Generation Systems through User Feedback Channel](https://mlsanity.com/paper/2408.04194/) (2024)
- [TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs](https://mlsanity.com/paper/2508.02455/) (2025)

## Abstract (arXiv)

Code completion is widely used by software developers to provide coding suggestions given a partially written code snippet. Apart from the traditional code completion methods, which only support single token completion at minimal positions, recent studies show the ability to provide longer code completion at more flexible positions. However, such frequently triggered and longer completion results reduce the overall precision as they generate more invalid results. Moreover, different studies are mostly incompatible with each other. Thus, it is vital to develop an ensemble framework that can combine results from multiple models to draw merits and offset defects of each model. This paper conducts a coding simulation to collect data from code context and different code completion models and then apply the data in two tasks. First, we introduce an acceptance model which can dynamically control whether to display completion results to the developer. It uses simulation features to predict whether correct results exist in the output of these models. Our best model reduces the percentage of false-positive completion from 55.09% to 17.44%. Second, we design a fusion ranking scheme that can automatically identify the priority of the completion results and reorder the candidates from multiple code completion models. This scheme is flexible in dealing with various models, regardless of the type or the length of their completion results. We integrate this ranking scheme with two frequency models and a GPT-2 styled language model, along with the acceptance model to yield 27.80% and 37.64% increase in TOP1 and TOP5 accuracy, respectively. In addition, we propose a new code completion evaluation metric, Benefit-Cost Ratio(BCR), taking into account the benefit of keystrokes saving and hidden cost of completion list browsing, which is closer to real coder experience scenario.