# Incorporating External Knowledge through Pre-training for Natural Language to Code Generation

arXiv: https://arxiv.org/abs/2004.09015 · HTML: https://mlsanity.com/paper/2004.09015/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2020.

## Built on by (most cited first, 30 of 36)

- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [DocPrompting: Generating Code by Retrieving the Docs](https://mlsanity.com/paper/2207.05987/) (2022)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [Natural Language to Code Translation with Execution](https://mlsanity.com/paper/2204.11454/) (2022)
- [SelfEvolve: A Code Evolution Framework via Large Language Models](https://mlsanity.com/paper/2306.02907/) (2023)
- [MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages](https://mlsanity.com/paper/2203.08388/) (2022)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages](https://mlsanity.com/paper/2204.09653/) (2022)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Python Code Generation by Asking Clarification Questions](https://mlsanity.com/paper/2212.09885/) (2022)
- [Shellcode_IA32: A Dataset for Automatic Shellcode Generation](https://mlsanity.com/paper/2104.13100/) (2021)
- [Source Code Summarization with Structural Relative Position Guided Transformer](https://mlsanity.com/paper/2202.06521/) (2022)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation](https://mlsanity.com/paper/2208.10091/) (2022)
- [Improving Tree-Structured Decoder Training for Code Generation via Mutual Learning](https://mlsanity.com/paper/2105.14796/) (2021)
- [Exploring Dynamic Selection of Branch Expansion Orders for Code Generation](https://mlsanity.com/paper/2106.00261/) (2021)
- [On the Effectiveness of Pretrained Models for API Learning](https://mlsanity.com/paper/2204.03498/) (2022)
- [Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation](https://mlsanity.com/paper/2106.04447/) (2021)
- [AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation](https://mlsanity.com/paper/2504.04220/) (2025)
- [Code Generation from Natural Language with Less Prior and More Monolingual Data](https://mlsanity.com/paper/2101.00259/) (2021)
- [A Neural Network Architecture for Program Understanding Inspired by Human Behaviors](https://mlsanity.com/paper/2206.04730/) (2022)
- [Natural Language-Guided Programming](https://mlsanity.com/paper/2108.05198/) (2021)
- [CoRI: Collective Relation Integration with Data Augmentation for Open Information Extraction](https://mlsanity.com/paper/2106.00793/) (2021)
- [A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective](https://mlsanity.com/paper/2202.04847/) (2022)
- [Code Generation for Unknown Libraries via Reading API Documentations](https://mlsanity.com/paper/2202.07806/) (2022)
- [Transformer with Tree-order Encoding for Neural Program Generation](https://mlsanity.com/paper/2206.13354/) (2022)
- [NL2CMD: An Updated Workflow for Natural Language to Bash Commands Translation](https://mlsanity.com/paper/2302.07845/) (2023)
- [Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation](https://mlsanity.com/paper/2203.15319/) (2022)
- [An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code](https://mlsanity.com/paper/2307.02443/) (2023)
- [Teach me how to Label: Labeling Functions from Natural Language with Text-to-text Transformers](https://mlsanity.com/paper/2101.07138/) (2021)

## Abstract (arXiv)

Open-domain code generation aims to generate code in a general-purpose programming language (such as Python) from natural language (NL) intents. Motivated by the intuition that developers usually retrieve resources on the web when writing code, we explore the effectiveness of incorporating two varieties of external knowledge into NL-to-code generation: automatically mined NL-code pairs from the online programming QA forum StackOverflow and programming language API documentation. Our evaluations show that combining the two sources with data augmentation and retrieval-based data re-sampling improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa. The code and resources are available at https://github.com/neulab/external-knowledge-codegen.