Forming Trees with Treeformers
Not on the map. Why?
Abstract (arXiv)
Human language is known to exhibit a nested, hierarchical structure, allowing us to form complex sentences out of smaller pieces. However, many state-of-the-art neural networks models such as Transformers have no explicit hierarchical structure in its architecture -- that is, they don't have an inductive bias toward hierarchical structure. Additionally, Transformers are known to perform poorly on compositional generalization tasks which require such structures. In this paper, we introduce Treeformer, a general-purpose encoder module inspired by the CKY algorithm which learns a composition operator and pooling function to construct hierarchical encodings for phrases and sentences. Our extensive experiments demonstrate the benefits of incorporating hierarchical structure into the Transformer and show significant improvements in compositional generalization as well as in downstream tasks such as machine translation, abstractive summarization, and various natural language understanding tasks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 14 Jul 2022.
Builds on
39 of 59 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2015A large annotated corpus for learning natural language inferencecited by 1,837
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
- 2018Neural Network Acceptability Judgmentscited by 952
27 more
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
- 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
- 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
- 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
- 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
- 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
- 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
- 2016Recurrent Neural Network Grammarscited by 109
- 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
- 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
- 2017Learning to Compose Task-Specific Tree Structurescited by 48
- 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
- 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
- 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
- 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
- 2021On Compositional Generalization of Neural Machine Translationcited by 34
- 2015Learning to Understand Phrases by Embedding the Dictionarycited by 33
- 2017Generative and Discriminative Text Classification with Recurrent Neural Networkscited by 30
- 2020Tree-structured Attention with Hierarchical Accumulationcited by 30
- 2017Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMscited by 23
- 2019Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Datacited by 20
- 2021R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modelingcited by 19
- 2020Heads-up! Unsupervised Constituency Parsing via Self-Attention Headscited by 12
- 2015Compositional Distributional Semantics with Long Short Term Memorycited by 9
- 2021Improved Latent Tree Induction with Distant Supervision via Span Constraintscited by 6
- 2017Towards Neural Machine Translation with Latent Tree Attentioncited by 2
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.