# Discrete Dictionary-based Decomposition Layer for Structured Representation Learning

arXiv: https://arxiv.org/abs/2406.06976 · HTML: https://mlsanity.com/paper/2406.06976/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Large Memory Layers with Product Keys](https://mlsanity.com/paper/1907.05242/) (2019)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [Learning Associative Inference Using Fast Weight Memory](https://mlsanity.com/paper/2011.07831/) (2020)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [Emergent Symbols through Binding in External Memory](https://mlsanity.com/paper/2012.14601/) (2020)
- [Memorize or generalize? Searching for a compositional RNN in a haystack](https://mlsanity.com/paper/1802.06467/) (2018)
- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [Discrete-Valued Neural Communication](https://mlsanity.com/paper/2107.02367/) (2021)
- [Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving](https://mlsanity.com/paper/1910.06611/) (2019)
- [Self-Attentive Associative Memory](https://mlsanity.com/paper/2002.03519/) (2020)
- [Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes](https://mlsanity.com/paper/1610.09027/) (2016)
- [Learning to Reason with Third-Order Tensor Products](https://mlsanity.com/paper/1811.12143/) (2018)
- [Learning Invariant Molecular Representation in Latent Discrete Space](https://mlsanity.com/paper/2310.14170/) (2023)
- [Discrete Key-Value Bottleneck](https://mlsanity.com/paper/2207.11240/) (2022)
- [Grounded Object Centric Learning](https://mlsanity.com/paper/2307.09437/) (2023)
- [Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization](https://mlsanity.com/paper/2106.01317/) (2021)

## Abstract (arXiv)

Neuro-symbolic neural networks have been extensively studied to integrate symbolic operations with neural networks, thereby improving systematic generalization. Specifically, Tensor Product Representation (TPR) framework enables neural networks to perform differentiable symbolic operations by encoding the symbolic structure of data within vector spaces. However, TPR-based neural networks often struggle to decompose unseen data into structured TPR representations, undermining their symbolic operations. To address this decomposition problem, we propose a Discrete Dictionary-based Decomposition (D3) layer designed to enhance the decomposition capabilities of TPR-based models. D3 employs discrete, learnable key-value dictionaries trained to capture symbolic features essential for decomposition operations. It leverages the prior knowledge acquired during training to generate structured TPR representations by mapping input data to pre-learned symbolic features within these dictionaries. D3 is a straightforward drop-in layer that can be seamlessly integrated into any TPR-based model without modifications. Our experimental results demonstrate that D3 significantly improves the systematic generalization of various TPR-based models while requiring fewer additional parameters. Notably, D3 outperforms baseline models on the synthetic task that demands the systematic decomposition of unseen combinatorial data.