# Attention-based Iterative Decomposition for Tensor Product Representation

arXiv: https://arxiv.org/abs/2406.01012 · HTML: https://mlsanity.com/paper/2406.01012/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [Tracking the World State with Recurrent Entity Networks](https://mlsanity.com/paper/1612.03969/) (2016)
- [Learning Associative Inference Using Fast Weight Memory](https://mlsanity.com/paper/2011.07831/) (2020)
- [An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learning](https://mlsanity.com/paper/2302.04419/) (2023)
- [Emergent Symbols through Binding in External Memory](https://mlsanity.com/paper/2012.14601/) (2020)
- [Metalearned Neural Memory](https://mlsanity.com/paper/1907.09720/) (2019)
- [Memorize or generalize? Searching for a compositional RNN in a haystack](https://mlsanity.com/paper/1802.06467/) (2018)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Fast and Slow Learning of Recurrent Independent Mechanisms](https://mlsanity.com/paper/2105.08710/) (2021)
- [Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving](https://mlsanity.com/paper/1910.06611/) (2019)

## Built on by (most cited first, 2 of 2)

- [Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions](https://mlsanity.com/paper/2605.09967/) (2026)
- [Discrete Dictionary-based Decomposition Layer for Structured Representation Learning](https://mlsanity.com/paper/2406.06976/) (2024)

## Abstract (arXiv)

In recent research, Tensor Product Representation (TPR) is applied for the systematic generalization task of deep neural networks by learning the compositional structure of data. However, such prior works show limited performance in discovering and representing the symbolic structure from unseen test data because their decomposition to the structural representations was incomplete. In this work, we propose an Attention-based Iterative Decomposition (AID) module designed to enhance the decomposition operations for the structured representations encoded from the sequential input data with TPR. Our AID can be easily adapted to any TPR-based model and provides enhanced systematic decomposition through a competitive attention mechanism between input features and structured representations. In our experiments, AID shows effectiveness by significantly improving the performance of TPR-based prior works on the series of systematic generalization tasks. Moreover, in the quantitative and qualitative evaluations, AID produces more compositional and well-bound structural representations than other works.