mlsanityData to 7 Oct 2026

Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns

arXiv 2310.01749 ↗First posted 3 Oct 2023cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Attention, specifically scaled dot-product attention, has proven effective for natural language, but it does not have a mechanism for handling hierarchical patterns of arbitrary nesting depth, which limits its ability to recognize certain syntactic structures. To address this shortcoming, we propose stack attention: an attention operator that incorporates stacks, inspired by their theoretical connections to context-free languages (CFLs). We show that stack attention is analogous to standard attention, but with a latent model of syntax that requires no syntactic supervision. We propose two variants: one related to deterministic pushdown automata (PDAs) and one based on nondeterministic PDAs, which allows transformers to recognize arbitrary CFLs. We show that transformers with stack attention are very effective at learning CFLs that standard transformers struggle on, achieving strong results on a CFL with theoretically maximal parsing difficulty. We also show that stack attention is more effective at natural language modeling under a constrained parameter budget, and we include results on machine translation.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

28 of 52 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Layer Normalizationcited by 5,579
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  9. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  10. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  11. 2016Recurrent Neural Network Grammarscited by 109
  12. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
16 more
  1. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  2. 2017Structured Attention Networkscited by 95
  3. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  4. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  5. 2015Learning to Transduce with Unbounded Memorycited by 54
  6. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  7. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  8. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  9. 2015Describing Multimedia Content using Attention-based Encoder--Decoder Networkscited by 32
  10. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  11. 2019Quantity doesn't buy quality syntax with neural language modelscited by 25
  12. 2021Structural Guidance for Transformer Language Modelscited by 25
  13. 2021Transformers Generalize Linearlycited by 19
  14. 2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
  15. 2020Learning Context-Free Languages with Nondeterministic Stack RNNscited by 15
  16. 2022The Surprising Computational Power of Nondeterministic Stack RNNscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Training Neural Networks as Recognizers of Formal Languagescited by 23
  2. 2024Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scalecited by 9
  3. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 7
  4. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  5. 2024Compositional Generalization Across Distributional Shifts with Sparse Tree Operationscited by 5
  6. 2024A Transformer with Stack Attentioncited by 2
  7. 2025Emergent Stack Representations in Modeling Counter Languages Using Transformerscited by 2
  8. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  9. 2025A Systematic Study of Compositional Syntactic Transformer Language Modelscited by 1
  10. 2025Bearing Syntactic Fruit with Stack-Augmented Neural Networkscited by 1
  11. 2026Grokking as Structural Inference: Transformers Need Bayesian Lottery Ticketscited by 1
  12. 2024Finding path and cycle counting formulae in graphs with Deep Reinforcement Learningcited by 0
5 more
  1. 2024Efficient transformer with reinforced position embedding for language modelscited by 0
  2. 2025StackTrans: From Large Language Model to Large Pushdown Automata Modelcited by 0
  3. 2026Rational Transductorscited by 0
  4. 2026Transcript-Managed Transformers: Monotone Multi-Agent Collapse and Universality with Two Pop-Enabled Transcriptscited by 0
  5. 2026Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languagescited by 0