# The Importance of Being Recurrent for Modeling Hierarchical Structure

arXiv: https://arxiv.org/abs/1803.03585 · HTML: https://mlsanity.com/paper/1803.03585/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2018.

## Built on by (most cited first, 30 of 42)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Self-Attention Networks Can Process Bounded Hierarchical Languages](https://mlsanity.com/paper/2105.11115/) (2021)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Grokking of Hierarchical Structure in Vanilla Transformers](https://mlsanity.com/paper/2305.18741/) (2023)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Refining Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/2104.09635/) (2021)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [Sparse Universal Transformer](https://mlsanity.com/paper/2310.07096/) (2023)
- [Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models](https://mlsanity.com/paper/2203.10326/) (2022)
- [Can LLMs facilitate interpretation of pre-trained language models?](https://mlsanity.com/paper/2305.13386/) (2023)
- [Modeling Hierarchical Structures with Continuous Recursive Neural Networks](https://mlsanity.com/paper/2106.06038/) (2021)
- [Evaluating Document Coherence Modelling](https://mlsanity.com/paper/2103.10133/) (2021)
- [On the Difficulty of Translating Free-Order Case-Marking Languages](https://mlsanity.com/paper/2107.06055/) (2021)
- [Can Interpretation Predict Behavior on Unseen Data?](https://mlsanity.com/paper/2507.06445/) (2025)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model](https://mlsanity.com/paper/2305.10845/) (2023)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Investigating Recurrent Transformers with Dynamic Halt](https://mlsanity.com/paper/2402.00976/) (2024)
- [Fine-grained Interpretation and Causation Analysis in Deep NLP Models](https://mlsanity.com/paper/2105.08039/) (2021)
- [Implicit N-grams Induced by Recurrence](https://mlsanity.com/paper/2205.02724/) (2022)
- [Efficient Beam Tree Recursion](https://mlsanity.com/paper/2307.10779/) (2023)
- [AI and Generative AI for Research Discovery and Summarization](https://mlsanity.com/paper/2401.06795/) (2024)
- [A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time](https://mlsanity.com/paper/2206.02440/) (2022)
- [Forming Trees with Treeformers](https://mlsanity.com/paper/2207.06960/) (2022)
- [Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers](https://mlsanity.com/paper/2301.06758/) (2023)
- [Can Edge Probing Tasks Reveal Linguistic Knowledge in QA Models?](https://mlsanity.com/paper/2109.07102/) (2021)
- [Vy=akarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages](https://mlsanity.com/paper/2103.00854/) (2021)
- [ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning](https://mlsanity.com/paper/2111.14034/) (2021)

## Abstract (arXiv)

Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks such as language modeling (Linzen et al., 2016) and neural machine translation (Shi et al., 2016). In contrast, the ability to model structured data with non-recurrent neural networks has received little attention despite their success in many NLP tasks (Gehring et al., 2017; Vaswani et al., 2017). In this work, we compare the two architectures---recurrent versus non-recurrent---with respect to their ability to model hierarchical structure and find that recurrency is indeed important for this purpose.