mlsanityData to 7 Oct 2026

Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale

arXiv 2203.00633 ↗First posted 1 Mar 2022cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which represents each sentence as a single vector harms perplexity on document-level language modeling, providing evidence that a different kind of memory mechanism -- one that is independent of composed syntactic representations -- plays an important role in current successful models of long text.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 55 of its 84 references that are in the map. First posted on arXiv 1 Mar 2022.

Builds on

55 of 84 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
38 more
  1. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  2. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  3. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  4. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  5. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  6. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  7. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  8. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  9. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  10. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  11. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  12. 2016Recurrent Neural Network Grammarscited by 109
  13. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  14. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  15. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  16. 2015Multi-task Sequence to Sequence Learningcited by 94
  17. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  18. 2014Grammar as a Foreign Languagecited by 70
  19. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  20. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  21. 2021Syntax-BERT: Improving Pre-trained Transformers with Syntax Treescited by 51
  22. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  23. 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 47
  24. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  25. 2020Transition-based Parsing with Stack-Transformerscited by 37
  26. 2016A Fast Unified Model for Parsing and Sentence Understandingcited by 34
  27. 2020Tree-structured Attention with Hierarchical Accumulationcited by 30
  28. 2016Tree-to-Sequence Attentional Neural Machine Translationcited by 28
  29. 2018Syntactic Scaffolds for Semantic Structurescited by 27
  30. 2021Structural Guidance for Transformer Language Modelscited by 25
  31. 2019Structural Supervision Improves Learning of Non-Local Grammatical Dependenciescited by 24
  32. 2019Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understandingcited by 20
  33. 2017Towards String-to-Tree Neural Machine Translationcited by 19
  34. 2020Syntactic Structure Distillation Pretraining For Bidirectional Encoderscited by 18
  35. 2017Learning to Parse and Translate Improves Neural Machine Translationcited by 16
  36. 2016What Do Recurrent Neural Network Grammars Learn About Syntax?cited by 15
  37. 2017Predicting Target Language CCG Supertags Improves Neural Machine Translationcited by 15
  38. 2019PaLM: A Hybrid Parser and Language Modelcited by 13

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 15; 2024: 11; 2025: 4; 2026: 7.

40 papers.

Sort:

By area:

  1. 2024Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?cited by 23
  2. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  3. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
  4. 2023A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?cited by 17
  5. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
  6. 2024Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scalecited by 9
  7. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 7
  8. 2023Differentiable Tree Operations Promote Compositional Generalizationcited by 5
  9. 2023Augmenting Transformers with Recursively Composed Multi-grained Representationscited by 5
  10. 2023Infusing Lattice Symmetry Priors in Attention Mechanisms for Sample-Efficient Abstract Geometric Reasoningcited by 4
  11. 2024Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervisioncited by 4
  12. 2022Forming Trees with Treeformerscited by 3
28 more
  1. 2023Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Buildingcited by 3
  2. 2024Punctuation Restoration Improves Structure Understanding Without Supervisioncited by 3
  3. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3
  4. 2024Sneaking Syntax into Transformer Language Models with Tree Regularizationcited by 3
  5. 2022Unbiased and Efficient Sampling of Dependency Treescited by 2
  6. 2023StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structurecited by 2
  7. 2023Polar Ducks and Where to Find Them: Enhancing Entity Linking with Duck Typing and Polar Box Embeddingscited by 2
  8. 2023SynJax: Structured Probability Distributions for JAXcited by 2
  9. 2024A Comparative Investigation of Compositional Syntax and Semantics in DALL-E 2cited by 2
  10. 2024A Transformer with Stack Attentioncited by 2
  11. 2022Composition, Attention, or Both?cited by 1
  12. 2023Improving Grammar-based Sequence-to-Sequence Modeling with Decomposition and Constraintscited by 1
  13. 2023The Quo Vadis of the Relationship between Language and Large Language Modelscited by 1
  14. 2023AMR Parsing with Causal Hierarchical Attention and Pointerscited by 1
  15. 2024Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situationscited by 1
  16. 2024Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddingscited by 1
  17. 2025A Systematic Study of Compositional Syntactic Transformer Language Modelscited by 1
  18. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  19. 2026A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Modelscited by 1
  20. 2023Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structurescited by 0
  21. 2025Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and Englishcited by 0
  22. 2025StackTrans: From Large Language Model to Large Pushdown Automata Modelcited by 0
  23. 2025Compositionality in Time Series: A Proof of Concept using Symbolic Dynamics and Compositional Data Augmentationcited by 0
  24. 2026Top-down string-to-dependency Neural Machine Translationcited by 0
  25. 2026GiLT: Augmenting Transformer Language Models with Dependency Graphscited by 0
  26. 2026Explaining Black-Box Language Models: Learning to Optimize Linguistically-Structured Word Subsetscited by 0
  27. 2026Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammarscited by 0
  28. 2026Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformerscited by 0