mlsanityData to 7 Oct 2026

Tree Transformer: Integrating Tree Structures into Self-Attention

arXiv 1909.06639 ↗First posted 14 Sep 2019cs.CL cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks. However, it is unclear what the learned attention captures. The attention computed by attention heads seems not to match human intuitions about hierarchical structures. This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures. The tree structures can be automatically induced from raw texts by our proposed "Constituent Attention" module, which is simply implemented by self-attention between two adjacent words. With the same training procedure identical to BERT, the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures, better language modeling, and further learning more explainable attention scores.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 10; 2023: 20; 2024: 10; 2025: 8; 2026: 5.

68 papers.

Sort:

By area:

  1. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  2. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  3. 2021Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualizationcited by 51
  4. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  5. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  6. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  7. 2021Link Prediction on N-ary Relational Facts: A Graph-based Approachcited by 29
  8. 2021Structural Guidance for Transformer Language Modelscited by 25
  9. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
  10. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
  11. 2021Auto-Parsing Network for Image Captioning and Visual Question Answeringcited by 16
  12. 2021Modeling Hierarchical Structures with Continuous Recursive Neural Networkscited by 15
38 more
  1. 2023Misalign, Contrast then Distill: Rethinking Misalignments in Language-Image Pretrainingcited by 10
  2. 2023Learning Trajectory-Word Alignments for Video-Language Taskscited by 7
  3. 2023Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parserscited by 7
  4. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 7
  5. 2022An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classificationcited by 6
  6. 2023Beam Tree Recursive Cellscited by 6
  7. 2024HDT: Hierarchical Document Transformercited by 6
  8. 2023Differentiable Tree Operations Promote Compositional Generalizationcited by 5
  9. 2024Compositional Generalization Across Distributional Shifts with Sparse Tree Operationscited by 5
  10. 2021Improving BERT Pretraining with Syntactic Supervisioncited by 4
  11. 2022An Interpretability Evaluation Benchmark for Pre-trained Language Modelscited by 4
  12. 2023Efficient Beam Tree Recursioncited by 4
  13. 2023Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representationcited by 4
  14. 2024Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervisioncited by 4
  15. 2021Co-training an Unsupervised Constituency Parser with Weak Supervisioncited by 3
  16. 2022Image Captioning In the Transformer Agecited by 3
  17. 2022RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generatorscited by 3
  18. 2022Transformer with Tree-order Encoding for Neural Program Generationcited by 3
  19. 2022Forming Trees with Treeformerscited by 3
  20. 2024Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Taskscited by 3
  21. 2024Tree Transformers are an Ineffective Model of Syntactic Constituencycited by 3
  22. 2024Sneaking Syntax into Transformer Language Models with Tree Regularizationcited by 3
  23. 2021RL-GRIT: Reinforcement Learning for Grammar Inferencecited by 2
  24. 2023StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structurecited by 2
  25. 2023Towards Early Prediction of Human iPSC Reprogramming Successcited by 2
  26. 2023Tree Cross Attentioncited by 2
  27. 2023Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Modelscited by 2
  28. 2021Enhancing the Transformer Decoder with Transition-based Syntaxcited by 1
  29. 2021ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoningcited by 1
  30. 2022Dependency-based Mixture Language Modelscited by 1
  31. 2023An Effective Method using Phrase Mechanism in Neural Machine Translationcited by 1
  32. 2024Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Inductioncited by 1
  33. 2024Representation Learning of Geometric Treescited by 1
  34. 2025Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nestingcited by 1
  35. 2025Bringing Structure to Naturalness: On the Naturalness of ASTscited by 1
  36. 2025A Systematic Study of Compositional Syntactic Transformer Language Modelscited by 1
  37. 2025Tree-Based Deep Learning for Ranking Symbolic Integration Algorithmscited by 1
  38. 2025Cross-modal Full-mode Fine-grained Alignment for Text-to-Image Person Retrievalcited by 1

50 most cited of 68 All 68 (JSON)