# Neural Sequence-to-grid Module for Learning Symbolic Rules

arXiv: https://arxiv.org/abs/2101.04921 · HTML: https://mlsanity.com/paper/2101.04921/

## Key facts

- Cited by 5 papers in the map (data through 13 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Neural GPUs Learn Algorithms](https://mlsanity.com/paper/1511.08228/) (2015)
- [Learning to Execute](https://mlsanity.com/paper/1410.4615/) (2014)
- [Neural Programmer-Interpreters](https://mlsanity.com/paper/1511.06279/) (2015)
- [Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets](https://mlsanity.com/paper/1503.01007/) (2015)
- [Neural Arithmetic Logic Units](https://mlsanity.com/paper/1808.00508/) (2018)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [Learning to Transduce with Unbounded Memory](https://mlsanity.com/paper/1506.02516/) (2015)
- [Metalearned Neural Memory](https://mlsanity.com/paper/1907.09720/) (2019)
- [Making Neural Programming Architectures Generalize via Recursion](https://mlsanity.com/paper/1704.06611/) (2017)
- [Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes](https://mlsanity.com/paper/1610.09027/) (2016)
- [Grid Long Short-Term Memory](https://mlsanity.com/paper/1507.01526/) (2015)
- [Number Sequence Prediction Problems for Evaluating Computational Powers of Neural Networks](https://mlsanity.com/paper/1805.07494/) (2018)
- [Attending to Mathematical Language with Transformers](https://mlsanity.com/paper/1812.02825/) (2018)

## Built on by (most cited first, 5 of 5)

- [A comparison of LSTM and GRU networks for learning symbolic sequences](https://mlsanity.com/paper/2107.02248/) (2021)
- [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023)
- [Transformers discover an elementary calculation system exploiting local attention and grid-like problem representation](https://mlsanity.com/paper/2207.02536/) (2022)
- [Learning to Compute Gr"obner Bases](https://mlsanity.com/paper/2311.12904/) (2023)
- [Neural Attention Memory](https://mlsanity.com/paper/2302.09422/) (2023)

## Abstract (arXiv)

Logical reasoning tasks over symbols, such as learning arithmetic operations and computer program evaluations, have become challenges to deep learning. In particular, even state-of-the-art neural networks fail to achieve textitout-of-distribution (OOD) generalization of symbolic reasoning tasks, whereas humans can easily extend learned symbolic rules. To resolve this difficulty, we propose a neural sequence-to-grid (seq2grid) module, an input preprocessor that automatically segments and aligns an input sequence into a grid. As our module outputs a grid via a novel differentiable mapping, any neural network structure taking a grid input, such as ResNet or TextCNN, can be jointly trained with our module in an end-to-end fashion. Extensive experiments show that neural networks having our module as an input preprocessor achieve OOD generalization on various arithmetic and algorithmic problems including number sequence prediction problems, algebraic word problems, and computer program evaluation problems while other state-of-the-art sequence transduction models cannot. Moreover, we verify that our module enhances TextCNN to solve the bAbI QA tasks without external memory.