mlsanityData to 7 Oct 2026

Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

arXiv 2407.17406 ↗First posted 24 Jul 2024cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at https://github.com/zhaoyd1/Dep_Transformer_Grammars.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

38 of 64 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  9. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  10. 2019Revealing the Dark Secrets of BERTcited by 300
  11. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  12. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
26 more
  1. 2016Recurrent Neural Network Grammarscited by 109
  2. 2017Structured Attention Networkscited by 95
  3. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  4. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  5. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  6. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  7. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  8. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  9. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  10. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  11. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  12. 2020Transition-based Parsing with Stack-Transformerscited by 37
  13. 2020StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modelingcited by 33
  14. 2020Tree-structured Attention with Hierarchical Accumulationcited by 30
  15. 2021Structural Guidance for Transformer Language Modelscited by 25
  16. 2017Effective Inference for Generative Neural Parsingcited by 24
  17. 2021Attention Can Reflect Syntactic Structure (If You Let It)cited by 22
  18. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
  19. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
  20. 2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
  21. 2019PaLM: A Hybrid Parser and Language Modelcited by 13
  22. 2021Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modelingcited by 10
  23. 2015Dependency Recurrent Neural Language Models for Sentence Completioncited by 9
  24. 2023Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representationcited by 4
  25. 2017Arc-swift: A Novel Transition System for Dependency Parsingcited by 2
  26. 2023AMR Parsing with Causal Hierarchical Attention and Pointerscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025A Systematic Study of Compositional Syntactic Transformer Language Modelscited by 1
  2. 2025Tree Matching Networks for Natural Language Inference: Parameter-Efficient Semantic Understanding via Dependency Parse Treescited by 0
  3. 2026Top-down string-to-dependency Neural Machine Translationcited by 0
  4. 2026GiLT: Augmenting Transformer Language Models with Dependency Graphscited by 0
  5. 2026Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammarscited by 0
  6. 2026Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformerscited by 0
  7. 2026Learning Sparse Decision Trees via Transformer Variational Auto-Encoderscited by 0