# On the Effectiveness of Pretrained Models for API Learning

arXiv: https://arxiv.org/abs/2204.03498 · HTML: https://mlsanity.com/paper/2204.03498/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Abstract Syntax Networks for Code Generation and Semantic Parsing](https://mlsanity.com/paper/1704.07535/) (2017)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [Latent Predictor Networks for Code Generation](https://mlsanity.com/paper/1603.06744/) (2016)
- [Cross-Lingual Natural Language Generation via Pre-Training](https://mlsanity.com/paper/1909.10481/) (2019)
- [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021)
- [TreeGen: A Tree-Based Transformer Architecture for Code Generation](https://mlsanity.com/paper/1911.09983/) (2019)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019)
- [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016)
- [ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation](https://mlsanity.com/paper/2001.11314/) (2020)
- [Incorporating External Knowledge through Pre-training for Natural Language to Code Generation](https://mlsanity.com/paper/2004.09015/) (2020)
- [Pre-trained Language Model Representations for Language Generation](https://mlsanity.com/paper/1903.09722/) (2019)
- [Program Synthesis and Semantic Parsing with Learned Code Idioms](https://mlsanity.com/paper/1906.10816/) (2019)
- [ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation](https://mlsanity.com/paper/2104.08006/) (2021)
- [Language Generation with Recurrent Generative Adversarial Networks without Pre-training](https://mlsanity.com/paper/1706.01399/) (2017)
- [Multimodal Representation for Neural Code Search](https://mlsanity.com/paper/2107.00992/) (2021)
- [SWIM: Synthesizing What I Mean](https://mlsanity.com/paper/1511.08497/) (2015)

## Built on by (most cited first, 9 of 9)

- [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [InstructExcel: A Benchmark for Natural Language Instruction in Excel](https://mlsanity.com/paper/2310.14495/) (2023)
- [Enhancing Automated Program Repair through Fine-tuning and Prompt Engineering](https://mlsanity.com/paper/2304.07840/) (2023)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?](https://mlsanity.com/paper/2309.07804/) (2023)
- [APIContext2Com: Code Comment Generation by Incorporating Pre-Defined API Documentation](https://mlsanity.com/paper/2303.01645/) (2023)
- [Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models](https://mlsanity.com/paper/2403.13588/) (2024)
- [A Lean Transformer Model for Dynamic Malware Analysis and Detection](https://mlsanity.com/paper/2408.02313/) (2024)

## Abstract (arXiv)

Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc. Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner. Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences. As it stands, the first approach treats queries and API names as bags of words. It lacks deep comprehension of the semantics of the queries. The latter approach adapts a neural language model to encode a user query into a fixed-length context vector and generate API sequences from the context vector. We want to understand the effectiveness of recent Pre-trained Transformer based Models (PTMs) for the API learning task. These PTMs are trained on large natural language corpora in an unsupervised manner to retain contextual knowledge about the language and have found success in solving similar Natural Language Processing (NLP) problems. However, the applicability of PTMs has not yet been explored for the API sequence generation task. We use a dataset that contains 7 million annotations collected from GitHub to evaluate the PTMs empirically. This dataset was also used to assess previous approaches. Based on our results, PTMs generate more accurate API sequences and outperform other related methods by around 11%. We have also identified two different tokenization approaches that can contribute to a significant boost in PTMs' performance for the API sequence generation task.