# Modeling Hierarchical Structures with Continuous Recursive Neural Networks

arXiv: https://arxiv.org/abs/2106.06038 · HTML: https://mlsanity.com/paper/2106.06038/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 74 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Theoretical Limitations of Self-Attention in Neural Sequence Models](https://mlsanity.com/paper/1906.06755/) (2019)
- [Neural Logic Machines](https://mlsanity.com/paper/1904.11694/) (2019)
- [ListOps: A Diagnostic Dataset for Latent Tree Learning](https://mlsanity.com/paper/1804.06028/) (2018)
- [Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks](https://mlsanity.com/paper/1810.09536/) (2018)
- [Structured Attention Networks](https://mlsanity.com/paper/1702.00887/) (2017)
- [Tree Transformer: Integrating Tree Structures into Self-Attention](https://mlsanity.com/paper/1909.06639/) (2019)
- [SparseMAP: Differentiable Sparse Structured Inference](https://mlsanity.com/paper/1802.04223/) (2018)
- [Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders](https://mlsanity.com/paper/1904.02142/) (2019)
- [Neural Language Modeling by Jointly Learning Syntax and Lexicon](https://mlsanity.com/paper/1711.02013/) (2017)
- [Learning to Transduce with Unbounded Memory](https://mlsanity.com/paper/1506.02516/) (2015)
- [Compositional Generalization by Learning Analytical Expressions](https://mlsanity.com/paper/2006.10627/) (2020)
- [Learning to Compose Task-Specific Tree Structures](https://mlsanity.com/paper/1707.02786/) (2017)
- [Unsupervised Recurrent Neural Network Grammars](https://mlsanity.com/paper/1904.03746/) (2019)
- [The Importance of Being Recurrent for Modeling Hierarchical Structure](https://mlsanity.com/paper/1803.03585/) (2018)
- [A Fast Unified Model for Parsing and Sentence Understanding](https://mlsanity.com/paper/1603.06021/) (2016)
- [StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling](https://mlsanity.com/paper/2012.00857/) (2020)
- [Learning to Compose Words into Sentences with Reinforcement Learning](https://mlsanity.com/paper/1611.09100/) (2016)
- [Tree-structured Attention with Hierarchical Accumulation](https://mlsanity.com/paper/2002.08046/) (2020)
- [Cooperative Learning of Disjoint Syntax and Semantics](https://mlsanity.com/paper/1902.09393/) (2019)
- [Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs](https://mlsanity.com/paper/1705.09189/) (2017)

## Built on by (most cited first, 15 of 15)

- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [Sliced Recursive Transformer](https://mlsanity.com/paper/2111.05297/) (2021)
- [Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representation](https://mlsanity.com/paper/2203.00281/) (2022)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Augmenting Transformers with Recursively Composed Multi-grained Representations](https://mlsanity.com/paper/2309.16319/) (2023)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Investigating Recurrent Transformers with Dynamic Halt](https://mlsanity.com/paper/2402.00976/) (2024)
- [A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classification](https://mlsanity.com/paper/2303.02860/) (2023)
- [Efficient Beam Tree Recursion](https://mlsanity.com/paper/2307.10779/) (2023)
- [Banyan: Improved Representation Learning with Explicit Structure](https://mlsanity.com/paper/2407.17771/) (2024)
- [Tree Transformers are an Ineffective Model of Syntactic Constituency](https://mlsanity.com/paper/2411.16993/) (2024)
- [On the Design Space Between Transformers and Recursive Neural Nets](https://mlsanity.com/paper/2409.01531/) (2024)
- [TRA: Better Length Generalisation with Threshold Relative Attention](https://mlsanity.com/paper/2503.23174/) (2025)
- [Interpreting Sentiment Composition with Latent Semantic Tree](https://mlsanity.com/paper/2308.16588/) (2023)
- [Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers](https://mlsanity.com/paper/2608.31067/) (2026)

## Abstract (arXiv)

Recursive Neural Networks (RvNNs), which compose sequences according to their underlying hierarchical syntactic structure, have performed well in several natural language processing tasks compared to similar models without structural biases. However, traditional RvNNs are incapable of inducing the latent structure in a plain text sequence on their own. Several extensions have been proposed to overcome this limitation. Nevertheless, these extensions tend to rely on surrogate gradients or reinforcement learning at the cost of higher bias or variance. In this work, we propose Continuous Recursive Neural Network (CRvNN) as a backpropagation-friendly alternative to address the aforementioned limitations. This is done by incorporating a continuous relaxation to the induced structure. We demonstrate that CRvNN achieves strong performance in challenging synthetic tasks such as logical inference and ListOps. We also show that CRvNN performs comparably or better than prior latent structure models on real-world tasks such as sentiment analysis and natural language inference.