# Maybe Deep Neural Networks are the Best Choice for Modeling Source Code

arXiv: https://arxiv.org/abs/1903.05734 · HTML: https://mlsanity.com/paper/1903.05734/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2019.

## Built on by (most cited first, 5 of 5)

- [Toward a Theory of Causation for Interpreting Neural Code Models](https://mlsanity.com/paper/2302.03788/) (2023)
- [Evaluating and Explaining Large Language Models for Code Using Syntactic Structures](https://mlsanity.com/paper/2308.03873/) (2023)
- [Improving Code Autocompletion with Transfer Learning](https://mlsanity.com/paper/2105.05991/) (2021)
- [Natural Language-Guided Programming](https://mlsanity.com/paper/2108.05198/) (2021)
- [Learning to Extend Program Graphs to Work-in-Progress Code](https://mlsanity.com/paper/2105.14038/) (2021)

## Abstract (arXiv)

Statistical language modeling techniques have successfully been applied to source code, yielding a variety of new software development tools, such as tools for code suggestion and improving readability. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. But traditional language models limit the vocabulary to a fixed set of common words. For code, this strong assumption has been shown to have a significant negative effect on predictive performance. But the open vocabulary version of the neural network language models for code have not been introduced in the literature. We present a new open-vocabulary neural language model for code that is not limited to a fixed vocabulary of identifier names. We employ a segmentation into subword units, subsequences of tokens chosen based on a compression criterion, following previous work in machine translation. Our network achieves best in class performance, outperforming even the state-of-the-art methods of Hellendoorn and Devanbu that are designed specifically to model code. Furthermore, we present a simple method for dynamically adapting the model to a new test project, resulting in increased performance. We showcase our methodology on code corpora in three different languages of over a billion tokens each, hundreds of times larger than in previous work. To our knowledge, this is the largest neural language model for code that has been reported.