A Transformer with Stack Attention
Not on the map. Why?
Abstract (arXiv)
Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 7 May 2024.
Builds on
34 of 56 references.
References · 34
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2016Layer Normalizationcited by 5,579
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2021Datasets: A Community Library for Natural Language Processingcited by 526
22 more
- 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
- 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
- 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
- 2022Neural Networks and the Chomsky Hierarchycited by 214
- 2021Thinking Like Transformerscited by 182
- 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
- 2016Recurrent Neural Network Grammarscited by 109
- 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
- 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
- 2022A Logic for Expressing Log-Precision Transformerscited by 67
- 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 65
- 2015Learning to Transduce with Unbounded Memorycited by 54
- 2020How Can Self-Attention Networks Recognize Dyck-n Languages?cited by 51
- 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
- 2023Logical Languages Accepted by Transformer Encoders with Hard Attentioncited by 37
- 2019Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languagescited by 31
- 2024Transformers Can Represent n-gram Language Modelscited by 31
- 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
- 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
- 2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
- 2020Learning Context-Free Languages with Nondeterministic Stack RNNscited by 15
- 2018Context-Free Transductions with Neural Stackscited by 14
Built on by
Citing papers per year: 2025: 2.
2 papers.