# What Do Recurrent Neural Network Grammars Learn About Syntax?

arXiv: https://arxiv.org/abs/1611.05774 · HTML: https://mlsanity.com/paper/1611.05774/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2016.

## Built on by (most cited first, 15 of 15)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)
- [Modeling structure-building in the brain with CCG parsing and large language models](https://mlsanity.com/paper/2210.16147/) (2022)
- [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024)
- [Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars](https://mlsanity.com/paper/2109.04939/) (2021)
- [Exploiting Emotion-Semantic Correlations for Empathetic Response Generation](https://mlsanity.com/paper/2402.17437/) (2024)
- [Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision](https://mlsanity.com/paper/2402.12691/) (2024)
- [If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?](https://mlsanity.com/paper/2502.11469/) (2025)
- [Head-driven Phrase Structure Parsing in O(n3) Time Complexity](https://mlsanity.com/paper/2105.09835/) (2021)
- [Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models](https://mlsanity.com/paper/2310.16240/) (2023)
- [In-Order Chart-Based Constituent Parsing](https://mlsanity.com/paper/2102.04065/) (2021)
- [Composition, Attention, or Both?](https://mlsanity.com/paper/2210.12958/) (2022)
- [Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion](https://mlsanity.com/paper/2204.08644/) (2022)
- [Representing and Parsing Korean Constituency Structure at Different Levels of Granularity](https://mlsanity.com/paper/2608.27035/) (2026)

## Abstract (arXiv)

Recurrent neural network grammars (RNNG) are a recently proposed probabilistic generative modeling family for natural language. They show state-of-the-art language modeling and parsing performance. We investigate what information they learn, from a linguistic perspective, through various ablations to the model and the data, and by augmenting the model with an attention mechanism (GA-RNNG) to enable closer inspection. We find that explicit modeling of composition is crucial for achieving the best performance. Through the attention mechanism, we find that headedness plays a central role in phrasal representation (with the model's latent attention largely agreeing with predictions made by hand-crafted head rules, albeit with some important differences). By training grammars without nonterminal labels, we find that phrasal representations depend minimally on nonterminals, providing support for the endocentricity hypothesis.