# Learning to Transduce with Unbounded Memory

arXiv: https://arxiv.org/abs/1506.02516 · HTML: https://mlsanity.com/paper/1506.02516/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2015.

## Built on by (most cited first, 30 of 54)

- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [Scaling Transformer to 1M tokens and beyond with RMT](https://mlsanity.com/paper/2304.11062/) (2023)
- [Personalized Federated Learning through Local Memorization](https://mlsanity.com/paper/2111.09360/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [ABC: Attention with Bounded-memory Control](https://mlsanity.com/paper/2110.02488/) (2021)
- [Lexicon Learning for Few-Shot Neural Sequence Modeling](https://mlsanity.com/paper/2106.03993/) (2021)
- [Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions](https://mlsanity.com/paper/2310.16076/) (2023)
- [Pushdown Layers: Encoding Recursive Structure in Transformer Language Models](https://mlsanity.com/paper/2310.19089/) (2023)
- [Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns](https://mlsanity.com/paper/2310.01749/) (2023)
- [Learning Hierarchical Structures with Differentiable Nondeterministic Stacks](https://mlsanity.com/paper/2109.01982/) (2021)
- [Modeling Hierarchical Structures with Continuous Recursive Neural Networks](https://mlsanity.com/paper/2106.06038/) (2021)
- [Autoregressive Large Language Models are Computationally Universal](https://mlsanity.com/paper/2410.03170/) (2024)
- [UnweaveNet: Unweaving Activity Stories](https://mlsanity.com/paper/2112.10194/) (2021)
- [A Call for Clarity in Beam Search: How It Works and When It Stops](https://mlsanity.com/paper/2204.05424/) (2022)
- [Formal Language Theory Meets Modern NLP](https://mlsanity.com/paper/2102.10094/) (2021)
- [Recognizing and Verifying Mathematical Equations using Multiplicative Differential Neural Units](https://mlsanity.com/paper/2104.02899/) (2021)
- [Learning to Navigate Intersections with Unsupervised Driver Trait Inference](https://mlsanity.com/paper/2109.06783/) (2021)
- [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://mlsanity.com/paper/2210.13569/) (2022)
- [On the Computational Complexity and Formal Hierarchy of Second Order Recurrent Neural Networks](https://mlsanity.com/paper/2309.14691/) (2023)
- [∞-former: Infinite Memory Transformer](https://mlsanity.com/paper/2109.00301/) (2021)
- [Neural Priority Queues for Graph Neural Networks](https://mlsanity.com/paper/2307.09660/) (2023)
- [The Surprising Computational Power of Nondeterministic Stack RNNs](https://mlsanity.com/paper/2210.01343/) (2022)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Neural Sequence-to-grid Module for Learning Symbolic Rules](https://mlsanity.com/paper/2101.04921/) (2021)
- [Evolutionary Training and Abstraction Yields Algorithmic Generalization of Neural Computers](https://mlsanity.com/paper/2105.07957/) (2021)
- [Differentiable Tree Operations Promote Compositional Generalization](https://mlsanity.com/paper/2306.00751/) (2023)
- [Compositional Generalization Across Distributional Shifts with Sparse Tree Operations](https://mlsanity.com/paper/2412.14076/) (2024)

## Abstract (arXiv)

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.