# Automatic Code Generation using Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2102.10535 · HTML: https://mlsanity.com/paper/2102.10535/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 11 references that are in the map.
- First posted on arXiv 21 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [TreeGen: A Tree-Based Transformer Architecture for Code Generation](https://mlsanity.com/paper/1911.09983/) (2019)
- [A Grammar-Based Structural CNN Decoder for Code Generation](https://mlsanity.com/paper/1811.06837/) (2018)
- [TreeGAN: Syntax-Aware Sequence Generation with Generative Adversarial Networks](https://mlsanity.com/paper/1808.07582/) (2018)
- [CodeGRU: Context-aware Deep Learning with Gated Recurrent Unit for Source Code Modeling](https://mlsanity.com/paper/1903.00884/) (2019)

## Built on by (most cited first, 8 of 8)

- [Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning](https://mlsanity.com/paper/2305.18869/) (2023)
- [Generating Computational Cognitive Models using Large Language Models](https://mlsanity.com/paper/2502.00879/) (2025)
- [Fine-tune Language Models to Approximate Unbiased In-context Learning](https://mlsanity.com/paper/2310.03331/) (2023)
- [LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops](https://mlsanity.com/paper/2603.23613/) (2026)
- [Test Code Generation for Telecom Software Systems using Two-Stage Generative Model](https://mlsanity.com/paper/2404.09249/) (2024)
- [Architecture and Knowledge Representation for Composable Inductive Programming](https://mlsanity.com/paper/2212.12320/) (2022)
- [Automated Creation of Source Code Variants of a Cryptographic Hash Function Implementation Using Generative Pre-Trained Transformer Models](https://mlsanity.com/paper/2404.15681/) (2024)
- [A Systematic Literature Review on the Use of Machine Learning in Software Engineering](https://mlsanity.com/paper/2406.13877/) (2024)

## Abstract (arXiv)

Recent advancements in natural language processing citegpt2 citeBERT have led to near-human performance in multiple natural language tasks. In this paper, we seek to understand whether similar techniques can be applied to a highly structured environment with strict syntax rules. Specifically, we propose an end-to-end machine learning model for code generation in the Python language built on-top of pre-trained language models. We demonstrate that a fine-tuned model can perform well in code generation tasks, achieving a BLEU score of 0.22, an improvement of 46% over a reasonable sequence-to-sequence baseline. All results and related code used for training and data processing are available on GitHub.