# A deep language model for software code

arXiv: https://arxiv.org/abs/1608.02715 · HTML: https://mlsanity.com/paper/1608.02715/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2016.

## Built on by (most cited first, 14 of 14)

- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python](https://mlsanity.com/paper/2201.08441/) (2022)
- [Programming Knowledge Tracing: A Comprehensive Dataset and A New Model](https://mlsanity.com/paper/2112.08273/) (2021)
- [MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation Learning](https://mlsanity.com/paper/2206.06460/) (2022)
- [A Neural Network Architecture for Program Understanding Inspired by Human Behaviors](https://mlsanity.com/paper/2206.04730/) (2022)
- [A Survey of Deep Learning Models for Structural Code Understanding](https://mlsanity.com/paper/2205.01293/) (2022)
- [Malicious Source Code Detection Using Transformer](https://mlsanity.com/paper/2209.07957/) (2022)
- [Autocompletion of Chief Complaints in the Electronic Health Records using Large Language Models](https://mlsanity.com/paper/2401.06088/) (2024)
- [deGraphCS: Embedding Variable-based Flow Graph for Neural Code Search](https://mlsanity.com/paper/2103.13020/) (2021)
- [PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis](https://mlsanity.com/paper/2306.00210/) (2023)
- [From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper](https://mlsanity.com/paper/2503.07450/) (2025)
- [Opportunities and Challenges of Applying Large Language Models in Building Energy Efficiency and Decarbonization Studies: An Exploratory Overview](https://mlsanity.com/paper/2312.11701/) (2023)
- [Bringing Structure to Naturalness: On the Naturalness of ASTs](https://mlsanity.com/paper/2504.08234/) (2025)
- [Multi-modal Learning for WebAssembly Reverse Engineering](https://mlsanity.com/paper/2404.03171/) (2024)

## Abstract (arXiv)

Existing language models such as n-grams for software code often fail to capture a long context where dependent code elements scatter far apart. In this paper, we propose a novel approach to build a language model for software code to address this particular issue. Our language model, partly inspired by human memory, is built upon the powerful deep learning-based Long Short Term Memory architecture that is capable of learning long-term dependencies which occur frequently in software code. Results from our intrinsic evaluation on a corpus of Java projects have demonstrated the effectiveness of our language model. This work contributes to realizing our vision for DeepSoft, an end-to-end, generic deep learning-based framework for modeling software and its development process.