mlsanityData to 7 Oct 2026

Modeling Hierarchical Structures with Continuous Recursive Neural Networks

arXiv 2106.06038 ↗First posted 10 Jun 2021cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Recursive Neural Networks (RvNNs), which compose sequences according to their underlying hierarchical syntactic structure, have performed well in several natural language processing tasks compared to similar models without structural biases. However, traditional RvNNs are incapable of inducing the latent structure in a plain text sequence on their own. Several extensions have been proposed to overcome this limitation. Nevertheless, these extensions tend to rely on surrogate gradients or reinforcement learning at the cost of higher bias or variance. In this work, we propose Continuous Recursive Neural Network (CRvNN) as a backpropagation-friendly alternative to address the aforementioned limitations. This is done by incorporating a continuous relaxation to the induced structure. We demonstrate that CRvNN achieves strong performance in challenging synthetic tasks such as logical inference and ListOps. We also show that CRvNN performs comparably or better than prior latent structure models on real-world tasks such as sentiment analysis and natural language inference.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 39 of its 74 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

39 of 74 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Layer Normalizationcited by 5,579
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2018Universal Transformerscited by 613
  9. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  10. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  11. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  12. 2019Neural Logic Machinescited by 144
27 more
  1. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  2. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  3. 2017Structured Attention Networkscited by 95
  4. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  5. 2018SparseMAP: Differentiable Sparse Structured Inferencecited by 62
  6. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  7. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  8. 2015Learning to Transduce with Unbounded Memorycited by 54
  9. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  10. 2017Learning to Compose Task-Specific Tree Structurescited by 48
  11. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  12. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  13. 2016A Fast Unified Model for Parsing and Sentence Understandingcited by 34
  14. 2020StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modelingcited by 33
  15. 2016Learning to Compose Words into Sentences with Reinforcement Learningcited by 30
  16. 2020Tree-structured Attention with Hierarchical Accumulationcited by 30
  17. 2019Cooperative Learning of Disjoint Syntax and Semanticscited by 27
  18. 2017Learning Structured Text Representationscited by 24
  19. 2020Retrofitting Structure-aware Transformer Language Model for End Taskscited by 24
  20. 2015Tree-structured composition in neural networks without tree-structured architecturescited by 23
  21. 2017Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMscited by 23
  22. 2019Ordered Memorycited by 20
  23. 2018On Tree-Based Neural Sentence Modelingcited by 16
  24. 2016Neural Tree Indexers for Text Understandingcited by 9
  25. 2018Latent Tree Learning with Differentiable Parsers: Shift-Reduce Parsing and Chart Parsingcited by 7
  26. 2019An Imitation Learning Approach to Unsupervised Parsingcited by 7
  27. 2019Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programmingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 1; 2023: 6; 2024: 4; 2025: 1; 2026: 1.

15 papers.

Sort:

  1. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  2. 2021Sliced Recursive Transformercited by 26
  3. 2022Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representationcited by 14
  4. 2023Beam Tree Recursive Cellscited by 6
  5. 2023Augmenting Transformers with Recursively Composed Multi-grained Representationscited by 5
  6. 2023Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalabilitycited by 5
  7. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  8. 2023A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classificationcited by 4
  9. 2023Efficient Beam Tree Recursioncited by 4
  10. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3
  11. 2024Tree Transformers are an Ineffective Model of Syntactic Constituencycited by 3
  12. 2024On the Design Space Between Transformers and Recursive Neural Netscited by 1
3 more
  1. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  2. 2023Interpreting Sentiment Composition with Latent Semantic Treecited by 0
  3. 2026Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformerscited by 0