mlsanityData to 7 Oct 2026

R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling

arXiv 2107.00967 ↗First posted 2 Jul 2021cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Human language understanding operates at multiple levels of granularity (e.g., words, phrases, and sentences) with increasing levels of abstraction that can be hierarchically combined. However, existing deep models with stacked layers do not explicitly model any sort of hierarchical process. This paper proposes a recursive Transformer model based on differentiable CKY style binary trees to emulate the composition process. We extend the bidirectional language model pre-training objective to this architecture, attempting to predict each word given its left and right abstraction nodes. To scale up our approach, we also introduce an efficient pruned tree induction algorithm to enable encoding in just a linear number of composition steps. Experimental results on language modeling and unsupervised parsing show the effectiveness of our approach.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

16 of 36 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  9. 2016Recurrent Neural Network Grammarscited by 109
  10. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  11. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  12. 2019Compound Probabilistic Context-Free Grammars for Grammar Inductioncited by 76
4 more
  1. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  2. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  3. 2016Learning to Compose Words into Sentences with Reinforcement Learningcited by 30
  4. 2017Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 6; 2024: 7; 2025: 1; 2026: 1.

19 papers.

Sort:

  1. 2022Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representationcited by 14
  2. 2024Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scalecited by 9
  3. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 8
  4. 2021Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentationcited by 6
  5. 2023Beam Tree Recursive Cellscited by 6
  6. 2023Augmenting Transformers with Recursively Composed Multi-grained Representationscited by 5
  7. 2023Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalabilitycited by 5
  8. 2023A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classificationcited by 4
  9. 2023Efficient Beam Tree Recursioncited by 4
  10. 2022Forming Trees with Treeformerscited by 3
  11. 2024Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Lesscited by 3
  12. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3
7 more
  1. 2024Sneaking Syntax into Transformer Language Models with Tree Regularizationcited by 3
  2. 2022Learning with Latent Structures in Natural Language Processing: A Surveycited by 2
  3. 2023AMR Parsing with Causal Hierarchical Attention and Pointerscited by 1
  4. 2026SymPlex: A Structure-Aware Transformer for Symbolic PDE Solvingcited by 1
  5. 2024Unsupervised Morphological Tree Tokenizercited by 0
  6. 2024On Eliciting Syntax from Language Models via Hashingcited by 0
  7. 2025Understanding Syntactic Generalization in Structure-inducing Language Modelscited by 0