# Grid Long Short-Term Memory

arXiv: https://arxiv.org/abs/1507.01526 · HTML: https://mlsanity.com/paper/1507.01526/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2015.

## Built on by (most cited first, 20 of 20)

- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [Learning the greatest common divisor: explaining transformer predictions](https://mlsanity.com/paper/2308.15594/) (2023)
- [Sequencer: Deep LSTM for Image Classification](https://mlsanity.com/paper/2205.01972/) (2022)
- [Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models](https://mlsanity.com/paper/2208.11445/) (2022)
- [Transformers discover an elementary calculation system exploiting local attention and grid-like problem representation](https://mlsanity.com/paper/2207.02536/) (2022)
- [Salsa Fresca: Angular Embeddings and Pre-Training for ML Attacks on Learning With Errors](https://mlsanity.com/paper/2402.01082/) (2024)
- [Neural Sequence-to-grid Module for Learning Symbolic Rules](https://mlsanity.com/paper/2101.04921/) (2021)
- [Embedding API Dependency Graph for Neural Code Generation](https://mlsanity.com/paper/2103.15361/) (2021)
- [An End-to-End Breast Tumour Classification Model Using Context-Based Patch Modelling- A BiLSTM Approach for Image Classification](https://mlsanity.com/paper/2106.02864/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Network Level Spatial Temporal Traffic State Forecasting with Hierarchical-Attention-LSTM (HierAttnLSTM)](https://mlsanity.com/paper/2201.05760/) (2022)
- [pLSTM: parallelizable Linear Source Transition Mark networks](https://mlsanity.com/paper/2506.11997/) (2025)
- [Gates Are Not What You Need in RNNs](https://mlsanity.com/paper/2108.00527/) (2021)
- [Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis](https://mlsanity.com/paper/2511.00989/) (2025)
- [R-MelNet: Reduced Mel-Spectral Modeling for Neural TTS](https://mlsanity.com/paper/2206.15276/) (2022)
- [Dilation-Erosion for Single-Frame Supervised Temporal Action Localization](https://mlsanity.com/paper/2212.06348/) (2022)
- [RigLSTM: Recurrent Independent Grid LSTM for Generalizable Sequence Learning](https://mlsanity.com/paper/2311.02123/) (2023)
- [Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents](https://mlsanity.com/paper/2610.02525/) (2026)

## Abstract (arXiv)

This paper introduces Grid Long Short-Term Memory, a network of LSTM cells arranged in a multidimensional grid that can be applied to vectors, sequences or higher dimensional data such as images. The network differs from existing deep LSTM architectures in that the cells are connected between network layers as well as along the spatiotemporal dimensions of the data. The network provides a unified way of using LSTM for both deep and sequential computation. We apply the model to algorithmic tasks such as 15-digit integer addition and sequence memorization, where it is able to significantly outperform the standard LSTM. We then give results for two empirical tasks. We find that 2D Grid LSTM achieves 1.47 bits per character on the Wikipedia character prediction benchmark, which is state-of-the-art among neural approaches. In addition, we use the Grid LSTM to define a novel two-dimensional translation model, the Reencoder, and show that it outperforms a phrase-based reference system on a Chinese-to-English translation task.