# RNNs Implicitly Implement Tensor Product Representations

arXiv: https://arxiv.org/abs/1812.08718 · HTML: https://mlsanity.com/paper/1812.08718/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2018.

## Built on by (most cited first, 26 of 26)

- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [On the proper role of linguistically-oriented deep net analysis in linguistic theorizing](https://mlsanity.com/paper/2106.08694/) (2021)
- [Characterizing Intrinsic Compositionality in Transformers with Tree Projections](https://mlsanity.com/paper/2211.01288/) (2022)
- [Relational Composition in Neural Networks: A Survey and Call to Action](https://mlsanity.com/paper/2407.14662/) (2024)
- [Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization](https://mlsanity.com/paper/2106.01317/) (2021)
- [Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models](https://mlsanity.com/paper/2210.03575/) (2022)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Quantum Mathematics in Artificial Intelligence](https://mlsanity.com/paper/2101.04255/) (2021)
- [Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages](https://mlsanity.com/paper/2208.06061/) (2022)
- [Differentiable Tree Operations Promote Compositional Generalization](https://mlsanity.com/paper/2306.00751/) (2023)
- [Compositional Generalization Across Distributional Shifts with Sparse Tree Operations](https://mlsanity.com/paper/2412.14076/) (2024)
- [Human-Like Geometric Abstraction in Large Pre-trained Neural Networks](https://mlsanity.com/paper/2402.04203/) (2024)
- [Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models](https://mlsanity.com/paper/2104.08410/) (2021)
- [Image interpretation by iterative bottom-up top-down processing](https://mlsanity.com/paper/2105.05592/) (2021)
- [Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings](https://mlsanity.com/paper/2301.08998/) (2023)
- [Semantic Composition in Visually Grounded Language Models](https://mlsanity.com/paper/2305.16328/) (2023)
- [Perturbation: A simple and efficient adversarial tracer for representation learning in language models](https://mlsanity.com/paper/2603.23821/) (2026)
- [Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions](https://mlsanity.com/paper/2605.09967/) (2026)
- [Probing of Quantitative Values in Abstractive Summarization Models](https://mlsanity.com/paper/2210.00667/) (2022)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)
- [Path Integration and Object-Location Binding Emerge in an Action-Conditioned Predictive Sequence Network](https://mlsanity.com/paper/2602.03490/) (2026)
- [A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability](https://mlsanity.com/paper/2608.29034/) (2026)
- [Binding Phoneme Identity to Position in Neural Models and the Human Brain](https://mlsanity.com/paper/2610.15843/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g., sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations. By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated by TPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations.