# Bringing Structure to Naturalness: On the Naturalness of ASTs

arXiv: https://arxiv.org/abs/2504.08234 · HTML: https://mlsanity.com/paper/2504.08234/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [Tree Transformer: Integrating Tree Structures into Self-Attention](https://mlsanity.com/paper/1909.06639/) (2019)
- [Structured Neural Summarization](https://mlsanity.com/paper/1811.01824/) (2018)
- [Large Language Models for Compiler Optimization](https://mlsanity.com/paper/2309.07062/) (2023)
- [On the "Naturalness" of Buggy Code](https://mlsanity.com/paper/1506.01159/) (2015)
- [Automatic Source Code Summarization with Extended Tree-LSTM](https://mlsanity.com/paper/1906.08094/) (2019)
- [Mining Idioms from Source Code](https://mlsanity.com/paper/1404.0417/) (2014)
- [A deep language model for software code](https://mlsanity.com/paper/1608.02715/) (2016)
- [AST-Transformer: Encoding Abstract Syntax Trees Efficiently for Code Summarization](https://mlsanity.com/paper/2112.01184/) (2021)
- [A deep tree-based model for software defect prediction](https://mlsanity.com/paper/1802.00921/) (2018)
- [Autofolding for Source Code Summarization](https://mlsanity.com/paper/1403.4503/) (2014)
- [Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?](https://mlsanity.com/paper/2401.01512/) (2024)
- [CodeBERT-nt: code naturalness via CodeBERT](https://mlsanity.com/paper/2208.06042/) (2022)

## Built on by (most cited first, 1 of 1)

- [Is Compression Really Linear with Code Intelligence?](https://mlsanity.com/paper/2505.11441/) (2025)

## Abstract (arXiv)

Source code comes in different shapes and forms. Previous research has already shown code to be more predictable than natural language as well as highlighted its statistical predictability at the token level: source code can be natural. More recently, the structure of code -- control flow, syntax graphs, abstract syntax trees etc. -- has been successfully used to improve the state-of-the-art on numerous tasks: code suggestion, code summarisation, method naming etc. This body of work implicitly assumes that structured representations of code are similarly statistically predictable, i.e. that a structured view of code is also natural. We consider that this view should be made explicit and propose directly studying the Structured Naturalness Hypothesis. Beyond just naming existing research that assumes this hypothesis and formulating it, we also provide evidence in the case of trees: TreeLSTM models over ASTs for some languages, such as Ruby, are competitive with n-gram models while handling the syntax token issue highlighted by previous research 'for free'. For other languages, such as Java or Python, we find tree models to perform worse, suggesting that downstream task improvement is uncorrelated to the language modelling task. Further, we show how such naturalness signals can be employed for near state-of-the-art results on just-in-time defect prediction while forgoing manual feature engineering work.