A Systematic Study of Compositional Syntactic Transformer Language Models
Not on the map. Why?
Abstract (arXiv)
Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This paper focuses on compositional SLMs that are based on constituency parse trees and contain explicit bottom-up composition of constituent representations. We identify key aspects of design choices in existing compositional SLMs and propose a unified framework encompassing both existing models and novel variants. We conduct a comprehensive empirical evaluation of all the variants in our framework across language modeling, syntactic generalization, summarization, dialogue, and inference efficiency. Based on the experimental results, we make multiple recommendations on the design of compositional SLMs. Our code is released at https://github.com/zhaoyd1/compositional_SLMs.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 28 Jun 2025.
Builds on
23 of 34 references.
References · 23
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
- 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
- 2016Recurrent Neural Network Grammarscited by 109
- 2017Structured Attention Networkscited by 95
- 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
- 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
11 more
- 2020Fast and Accurate Neural CRF Constituency Parsingcited by 50
- 2019Unsupervised Recurrent Neural Network Grammarscited by 47
- 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 41
- 2020StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modelingcited by 33
- 2017Effective Inference for Generative Neural Parsingcited by 25
- 2021Structural Guidance for Transformer Language Modelscited by 25
- 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 19
- 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
- 2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
- 2024Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scalecited by 10
- 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.