# Learning to Reason with Third-Order Tensor Products

arXiv: https://arxiv.org/abs/1811.12143 · HTML: https://mlsanity.com/paper/1811.12143/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2018.

## Built on by (most cited first, 27 of 27)

- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [Learning with Holographic Reduced Representations](https://mlsanity.com/paper/2109.02157/) (2021)
- [Recasting Self-Attention with Holographic Reduced Representations](https://mlsanity.com/paper/2305.19534/) (2023)
- [Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization](https://mlsanity.com/paper/2106.01317/) (2021)
- [Consecutive Batch Model Editing with HooK Layers](https://mlsanity.com/paper/2403.05330/) (2024)
- [Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures](https://mlsanity.com/paper/2401.16024/) (2024)
- [Towards Learning Abductive Reasoning using VSA Distributed Representations](https://mlsanity.com/paper/2406.19121/) (2024)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages](https://mlsanity.com/paper/2208.06061/) (2022)
- [Differentiable Tree Operations Promote Compositional Generalization](https://mlsanity.com/paper/2306.00751/) (2023)
- [Meta-Reinforcement Learning with Self-Modifying Networks](https://mlsanity.com/paper/2202.02363/) (2022)
- [Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks](https://mlsanity.com/paper/2212.14374/) (2022)
- [Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations](https://mlsanity.com/paper/2312.15310/) (2023)
- [A Walsh Hadamard Derived Linear Vector Symbolic Architecture](https://mlsanity.com/paper/2410.22669/) (2024)
- [DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text](https://mlsanity.com/paper/2310.12557/) (2023)
- [Attention-based Iterative Decomposition for Tensor Product Representation](https://mlsanity.com/paper/2406.01012/) (2024)
- [Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks](https://mlsanity.com/paper/2211.09440/) (2022)
- [Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions](https://mlsanity.com/paper/2605.09967/) (2026)
- [PAC-learning gains of Turing machines over circuits and neural networks](https://mlsanity.com/paper/2103.12686/) (2021)
- [Discrete Dictionary-based Decomposition Layer for Structured Representation Learning](https://mlsanity.com/paper/2406.06976/) (2024)
- [Who invented deep residual learning?](https://mlsanity.com/paper/2509.24732/) (2025)
- [Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling](https://mlsanity.com/paper/2609.36529/) (2026)

## Abstract (arXiv)

We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data. This improves symbolic interpretation and systematic generalisation. Our architecture is trained end-to-end through gradient descent on a variety of simple natural language reasoning tasks, significantly outperforming the latest state-of-the-art models in single-task and all-tasks settings. We also augment a subset of the data such that training and test data exhibit large systematic differences and show that our approach generalises better than the previous state-of-the-art.