mlsanityData to 7 Oct 2026

Automatic Code Generation using Pre-Trained Language Models

arXiv 2102.10535 ↗First posted 21 Feb 2021cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent advancements in natural language processing citegpt2 citeBERT have led to near-human performance in multiple natural language tasks. In this paper, we seek to understand whether similar techniques can be applied to a highly structured environment with strict syntax rules. Specifically, we propose an end-to-end machine learning model for code generation in the Python language built on-top of pre-trained language models. We demonstrate that a fine-tuned model can perform well in code generation tasks, achieving a BLEU score of 0.22, an improvement of 46% over a reasonable sequence-to-sequence baseline. All results and related code used for training and data processing are available on GitHub.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 10 of its 11 references that are in the map. First posted on arXiv 21 Feb 2021.

Builds on

10 of 11 references.

References · 10

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  6. 2015Character-Aware Neural Language Modelscited by 129
  7. 2019TreeGen: A Tree-Based Transformer Architecture for Code Generationcited by 58
  8. 2018A Grammar-Based Structural CNN Decoder for Code Generationcited by 29
  9. 2018TreeGAN: Syntax-Aware Sequence Generation with Generative Adversarial Networkscited by 6
  10. 2019CodeGRU: Context-aware Deep Learning with Gated Recurrent Unit for Source Code Modelingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 3; 2025: 1; 2026: 1.

8 papers.

  1. 2023Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learningcited by 19
  2. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  3. 2025Generating Computational Cognitive Models using Large Language Modelscited by 12
  4. 2026LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loopscited by 8
  5. 2024Test Code Generation for Telecom Software Systems using Two-Stage Generative Modelcited by 2
  6. 2022Architecture and Knowledge Representation for Composable Inductive Programmingcited by 1
  7. 2024Automated Creation of Source Code Variants of a Cryptographic Hash Function Implementation Using Generative Pre-Trained Transformer Modelscited by 0
  8. 2024A Systematic Literature Review on the Use of Machine Learning in Software Engineeringcited by 0