# Tree-structured Attention with Hierarchical Accumulation

arXiv: https://arxiv.org/abs/2002.08046 · HTML: https://mlsanity.com/paper/2002.08046/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 30 of 30)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization](https://mlsanity.com/paper/2110.10834/) (2021)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Modeling Hierarchical Structures with Continuous Recursive Neural Networks](https://mlsanity.com/paper/2106.06038/) (2021)
- [Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models](https://mlsanity.com/paper/2407.17406/) (2024)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [SLOTH: Structured Learning and Task-based Optimization for Time Series Forecasting on Hierarchies](https://mlsanity.com/paper/2302.05650/) (2023)
- [SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluation](https://mlsanity.com/paper/2206.10029/) (2022)
- [Efficient Beam Tree Recursion](https://mlsanity.com/paper/2307.10779/) (2023)
- [Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision](https://mlsanity.com/paper/2402.12691/) (2024)
- [GN-Transformer: Fusing Sequence and Graph Representation for Improved Code Summarization](https://mlsanity.com/paper/2111.08874/) (2021)
- [A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective](https://mlsanity.com/paper/2202.04847/) (2022)
- [RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generators](https://mlsanity.com/paper/2205.12590/) (2022)
- [Transformer with Tree-order Encoding for Neural Program Generation](https://mlsanity.com/paper/2206.13354/) (2022)
- [Forming Trees with Treeformers](https://mlsanity.com/paper/2207.06960/) (2022)
- [An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks](https://mlsanity.com/paper/2210.13002/) (2022)
- [Tree Cross Attention](https://mlsanity.com/paper/2309.17388/) (2023)
- [Enhancing the Transformer Decoder with Transition-based Syntax](https://mlsanity.com/paper/2101.12640/) (2021)
- [Graph Conditioned Sparse-Attention for Improved Source Code Understanding](https://mlsanity.com/paper/2112.00663/) (2021)
- [Continuous Decomposition of Granularity for Neural Paraphrase Generation](https://mlsanity.com/paper/2209.01765/) (2022)
- [Generic Dependency Modeling for Multi-Party Conversation](https://mlsanity.com/paper/2302.10680/) (2023)
- [An Effective Method using Phrase Mechanism in Neural Machine Translation](https://mlsanity.com/paper/2308.10482/) (2023)
- [STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants](https://mlsanity.com/paper/2310.16990/) (2023)
- [PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure](https://mlsanity.com/paper/2404.13645/) (2024)
- [Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nesting](https://mlsanity.com/paper/2502.15819/) (2025)
- [Hierarchy-Guided Topology Latent Flow for Molecular Graph Generation](https://mlsanity.com/paper/2603.27113/) (2026)
- [Mahalanobis-Based Multi-Head Attention for Complex State Propagation](https://mlsanity.com/paper/2608.24462/) (2026)

## Abstract (arXiv)

Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks. However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently. On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efficiently as the Transformer. In this paper, we attempt to bridge this gap with "Hierarchical Accumulation" to encode parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task. It also yields improvements over Transformer and Tree-LSTM on three text classification tasks. We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.