# Structured Generative Models of Natural Source Code

arXiv: https://arxiv.org/abs/1401.0514 · HTML: https://mlsanity.com/paper/1401.0514/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2014.

## Built on by (most cited first, 19 of 19)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [A Survey on Deep Graph Generation: Methods and Applications](https://mlsanity.com/paper/2203.06714/) (2022)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [Large Language Model-Aware In-Context Learning for Code Generation](https://mlsanity.com/paper/2310.09748/) (2023)
- [Controlling Conditional Language Models without Catastrophic Forgetting](https://mlsanity.com/paper/2112.00791/) (2021)
- [Learning to Complete Code with Sketches](https://mlsanity.com/paper/2106.10158/) (2021)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Neural Program Generation Modulo Static Analysis](https://mlsanity.com/paper/2111.01633/) (2021)
- [Energy-Based Models for Code Generation under Compilability Constraints](https://mlsanity.com/paper/2106.04985/) (2021)
- [Learning Intermediate Representations using Graph Neural Networks for NUMA and Prefetchers Optimization](https://mlsanity.com/paper/2203.00611/) (2022)
- [Efficacy of Various Large Language Models in Generating Smart Contracts](https://mlsanity.com/paper/2407.11019/) (2024)
- [Learning to Extend Program Graphs to Work-in-Progress Code](https://mlsanity.com/paper/2105.14038/) (2021)
- [Towards a Neural Lambda Calculus: Neurosymbolic AI Applied to the Foundations of Functional Programming](https://mlsanity.com/paper/2304.09276/) (2023)
- [Learning UI-to-Code Reverse Generator Using Visual Critic Without Rendering](https://mlsanity.com/paper/2305.14637/) (2023)
- [Aligning language models with human preferences](https://mlsanity.com/paper/2404.12150/) (2024)
- [Call Me Maybe: Enhancing JavaScript Call Graph Construction using Graph Neural Networks](https://mlsanity.com/paper/2506.18191/) (2025)
- [Serenity: Library Based Python Code Analysis for Code Completion and Automated Machine Learning](https://mlsanity.com/paper/2301.05108/) (2023)
- [A Study of Variable-Role-based Feature Enrichment in Neural Models of Code](https://mlsanity.com/paper/2303.04942/) (2023)

## Abstract (arXiv)

We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second, we learn a distributed representation of source code elements. Finally, they integrate closely with a compiler, which allows leveraging compiler logic and abstractions when building structure into the model. We also develop an extension that includes more complex structure, refining how the model generates identifier tokens based on what variables are currently in scope. Our models can be learned efficiently, and we show empirically that including appropriate structure greatly improves the models, measured by the probability of generating test programs.