# Structural Guidance for Transformer Language Models

arXiv: https://arxiv.org/abs/2108.00104 · HTML: https://mlsanity.com/paper/2108.00104/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Tree Transformer: Integrating Tree Structures into Self-Attention](https://mlsanity.com/paper/1909.06639/) (2019)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://mlsanity.com/paper/1909.02597/) (2019)
- [Do Syntax Trees Help Pre-trained Transformers Extract Information?](https://mlsanity.com/paper/2008.09084/) (2020)
- [Transition-based Parsing with Stack-Transformers](https://mlsanity.com/paper/2010.10669/) (2020)
- [Syntactic Scaffolds for Semantic Structures](https://mlsanity.com/paper/1808.10485/) (2018)
- [Improving BERT with Syntax-aware Local Attention](https://mlsanity.com/paper/2012.15150/) (2020)
- [Effective Inference for Generative Neural Parsing](https://mlsanity.com/paper/1707.08976/) (2017)
- [Structural Supervision Improves Learning of Non-Local Grammatical Dependencies](https://mlsanity.com/paper/1903.00943/) (2019)
- [Syntactic Structure Distillation Pretraining For Bidirectional Encoders](https://mlsanity.com/paper/2005.13482/) (2020)
- [PaLM: A Hybrid Parser and Language Model](https://mlsanity.com/paper/1909.02134/) (2019)
- [SG-Net: Syntax Guided Transformer for Language Representation](https://mlsanity.com/paper/2012.13915/) (2020)
- [Syntax-Directed Attention for Neural Machine Translation](https://mlsanity.com/paper/1711.04231/) (2017)
- [Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach](https://mlsanity.com/paper/2005.05864/) (2020)
- [Stack-propagation: Improved Representation Learning for Syntax](https://mlsanity.com/paper/1603.06598/) (2016)
- [Syntax-aware Multilingual Semantic Role Labeling](https://mlsanity.com/paper/1909.00310/) (2019)
- [Transformer-Based Neural Text Generation with Syntactic Guidance](https://mlsanity.com/paper/2010.01737/) (2020)
- [Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle](https://mlsanity.com/paper/2011.07960/) (2020)

## Built on by (most cited first, 25 of 25)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing](https://mlsanity.com/paper/2110.15534/) (2021)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [Pushdown Layers: Encoding Recursive Structure in Transformer Language Models](https://mlsanity.com/paper/2310.19089/) (2023)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns](https://mlsanity.com/paper/2310.01749/) (2023)
- [RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization](https://mlsanity.com/paper/2405.00657/) (2024)
- [Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling](https://mlsanity.com/paper/2112.07874/) (2021)
- [Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models](https://mlsanity.com/paper/2407.17406/) (2024)
- [Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models](https://mlsanity.com/paper/2109.11058/) (2021)
- [Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision](https://mlsanity.com/paper/2402.12691/) (2024)
- [Sneaking Syntax into Transformer Language Models with Tree Regularization](https://mlsanity.com/paper/2411.18885/) (2024)
- [Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models](https://mlsanity.com/paper/2310.16240/) (2023)
- [Infusing Prompts with Syntax and Semantics](https://mlsanity.com/paper/2412.06107/) (2024)
- [Composition, Attention, or Both?](https://mlsanity.com/paper/2210.12958/) (2022)
- [AMR Parsing with Causal Hierarchical Attention and Pointers](https://mlsanity.com/paper/2310.11964/) (2023)
- [Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations](https://mlsanity.com/paper/2401.14212/) (2024)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [A Systematic Study of Compositional Syntactic Transformer Language Models](https://mlsanity.com/paper/2506.22978/) (2025)
- [GiLT: Augmenting Transformer Language Models with Dependency Graphs](https://mlsanity.com/paper/2605.15562/) (2026)
- [Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures](https://mlsanity.com/paper/2305.18915/) (2023)
- [Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spaces](https://mlsanity.com/paper/2407.01310/) (2024)
- [Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars](https://mlsanity.com/paper/2606.16836/) (2026)
- [Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers](https://mlsanity.com/paper/2608.06111/) (2026)

## Abstract (arXiv)

Transformer-based language models pre-trained on large amounts of text data have proven remarkably successful in learning generic transferable linguistic representations. Here we study whether structural guidance leads to more human-like systematic linguistic generalization in Transformer language models without resorting to pre-training on very large amounts of data. We explore two general ideas. The "Generative Parsing" idea jointly models the incremental parse and word sequence as part of the same sequence modeling task. The "Structural Scaffold" idea guides the language model's representation via additional structure loss that separately predicts the incremental constituency parse. We train the proposed models along with a vanilla Transformer language model baseline on a 14 million-token and a 46 million-token subset of the BLLIP dataset, and evaluate models' syntactic generalization performances on SG Test Suites and sized BLiMP. Experiment results across two benchmarks suggest converging evidence that generative structural supervisions can induce more robust and humanlike linguistic generalization in Transformer language models without the need for data intensive pre-training.