# Self-Attentive Associative Memory

arXiv: https://arxiv.org/abs/2002.03519 · HTML: https://mlsanity.com/paper/2002.03519/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 30 of 33)

- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [ABC: Attention with Bounded-memory Control](https://mlsanity.com/paper/2110.02488/) (2021)
- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](https://mlsanity.com/paper/2204.13154/) (2022)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Recasting Self-Attention with Holographic Reduced Representations](https://mlsanity.com/paper/2305.19534/) (2023)
- [A Call for Clarity in Beam Search: How It Works and When It Stops](https://mlsanity.com/paper/2204.05424/) (2022)
- [Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception](https://mlsanity.com/paper/2308.05822/) (2023)
- [Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning](https://mlsanity.com/paper/2410.10132/) (2024)
- [Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures](https://mlsanity.com/paper/2508.10824/) (2025)
- [Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture](https://mlsanity.com/paper/2310.03052/) (2023)
- [Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking](https://mlsanity.com/paper/2112.00086/) (2021)
- [Learning to Rehearse in Long Sequence Memorization](https://mlsanity.com/paper/2106.01096/) (2021)
- [Short-Term Plasticity Neurons Learning to Learn and Forget](https://mlsanity.com/paper/2206.14048/) (2022)
- [Graph Neural Networks Extract High-Resolution Cultivated Land Maps from Sentinel-2 Image Series](https://mlsanity.com/paper/2208.02349/) (2022)
- [HIT: A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation](https://mlsanity.com/paper/2105.14600/) (2021)
- [Episodic Policy Gradient Training](https://mlsanity.com/paper/2112.01853/) (2021)
- [Learning Language Representations with Logical Inductive Bias](https://mlsanity.com/paper/2302.09458/) (2023)
- [Persona-aware Generative Model for Code-mixed Language](https://mlsanity.com/paper/2309.02915/) (2023)
- [DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text](https://mlsanity.com/paper/2310.12557/) (2023)
- [A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer](https://mlsanity.com/paper/2204.12753/) (2022)
- [Memory-Augmented Graph Neural Networks: A Brain-Inspired Review](https://mlsanity.com/paper/2209.10818/) (2022)
- [A Brain-inspired Memory Transformation based Differentiable Neural Computer for Reasoning-based Question Answering](https://mlsanity.com/paper/2301.02809/) (2023)
- [A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information](https://mlsanity.com/paper/2305.07565/) (2023)
- [Knowledge Authoring for Rules and Actions](https://mlsanity.com/paper/2305.07763/) (2023)
- [A Framework for Inference Inspired by Human Memory Mechanisms](https://mlsanity.com/paper/2310.09297/) (2023)
- [Attention-based Iterative Decomposition for Tensor Product Representation](https://mlsanity.com/paper/2406.01012/) (2024)
- [Re:Frame -- Retrieving Experience From Associative Memory](https://mlsanity.com/paper/2508.19344/) (2025)
- [Bird-Eye Transformers for Text Generation Models](https://mlsanity.com/paper/2210.03985/) (2022)

## Abstract (arXiv)

Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory). The idea is implemented through a novel Self-attentive Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of associative memories that represent the hypothetical high-order relationships between arbitrary pairs of memory elements, through which a relational memory is constructed from an item memory. The two memories are wired into a single sequential model capable of both memorization and relational reasoning. We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks, from challenging synthetic problems to practical testbeds such as geometry, graph, reinforcement learning, and question answering.