# Bearing Syntactic Fruit with Stack-Augmented Neural Networks

arXiv: https://arxiv.org/abs/2511.03547 · HTML: https://mlsanity.com/paper/2511.03547/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks](https://mlsanity.com/paper/1810.09536/) (2018)
- [Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets](https://mlsanity.com/paper/1503.01007/) (2015)
- [Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks](https://mlsanity.com/paper/2001.03632/) (2020)
- [Grokking of Hierarchical Structure in Vanilla Transformers](https://mlsanity.com/paper/2305.18741/) (2023)
- [Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks](https://mlsanity.com/paper/1802.09091/) (2018)
- [Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models](https://mlsanity.com/paper/2203.09397/) (2022)
- [Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers](https://mlsanity.com/paper/2404.16367/) (2024)
- [Training Neural Networks as Recognizers of Formal Languages](https://mlsanity.com/paper/2411.07107/) (2024)
- [How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases](https://mlsanity.com/paper/2305.19905/) (2023)
- [How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech](https://mlsanity.com/paper/2301.11462/) (2023)
- [In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax](https://mlsanity.com/paper/2311.07811/) (2023)
- [Transformers Generalize Linearly](https://mlsanity.com/paper/2109.12036/) (2021)
- [Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns](https://mlsanity.com/paper/2310.01749/) (2023)
- [Learning Hierarchical Structures with Differentiable Nondeterministic Stacks](https://mlsanity.com/paper/2109.01982/) (2021)
- [Learning Context-Free Languages with Nondeterministic Stack RNNs](https://mlsanity.com/paper/2010.04674/) (2020)
- [The Surprising Computational Power of Nondeterministic Stack RNNs](https://mlsanity.com/paper/2210.01343/) (2022)
- [Tree Transformers are an Ineffective Model of Syntactic Constituency](https://mlsanity.com/paper/2411.16993/) (2024)

## Built on by (most cited first, 1 of 1)

- [Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages](https://mlsanity.com/paper/2610.02040/) (2026)

## Abstract (arXiv)

When children learn language, they make syntactic generalizations based on hierarchical rules. A recent line of work has inquired as to whether common neural network architectures share this inductive bias for hierarchical syntax, finding that they do so only under special conditions: when augmented with ground-truth parse tree structures, when pre-trained on massive corpora, or when trained long past convergence. In this paper, we demonstrate, for the first time, neural network architectures that generalize in human-like fashion when trained only on surface forms: stack-augmented neural networks. We test three base architectures (transformer, simple RNN, LSTM) augmented with two styles of stack, one of which leverages nondeterminism. We find that transformers with nondeterministic stacks generalize best on multiple tasks designed to measure hierarchical inductive bias. This suggests that stack-augmented neural networks may be more accurate models of human syntax acquisition than standard architectures, serving as useful objects of psycholinguistic study. Our code is publicly available.