# MeMo: Towards Language Models with Associative Memory Mechanisms

arXiv: https://arxiv.org/abs/2502.12851 · HTML: https://mlsanity.com/paper/2502.12851/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [Memorization Capacity of Multi-Head Attention in Transformers](https://mlsanity.com/paper/2306.02010/) (2023)
- [Unveiling the Generalization Power of Fine-Tuned Large Language Models](https://mlsanity.com/paper/2403.09162/) (2024)
- [How BPE Affects Memorization in Transformers](https://mlsanity.com/paper/2110.02782/) (2021)
- [Position: Understanding LLMs Requires More Than Statistical Generalization](https://mlsanity.com/paper/2405.01964/) (2024)
- [The Dark Side of the Language: Pre-trained Transformers in the DarkNet](https://mlsanity.com/paper/2201.05613/) (2022)

## Built on by (most cited first, 5 of 5)

- [Neuro-symbolic Syntactic Parsing: Shaping a Neural Network with the CYK Algorithm](https://mlsanity.com/paper/2605.31421/) (2026)
- [When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMs](https://mlsanity.com/paper/2606.21203/) (2026)
- [Towards Version-aware Operations and Transaction Memories for Multi-layer MeMo](https://mlsanity.com/paper/2606.24040/) (2026)
- [LifeFuse-Mem: Lifecycle-Aware State Fusion Against Temporary Overwriting for Long-Term Memory](https://mlsanity.com/paper/2609.12436/) (2026)
- [Single-Layer MeMo as a Randomized Hamming-Kernel Classifier](https://mlsanity.com/paper/2609.34562/) (2026)

## Abstract (arXiv)

Memorization is a fundamental ability of Transformer-based Large Language Models, achieved through learning. In this paper, we propose a paradigm shift by designing an architecture to memorize text directly, bearing in mind the principle that memorization precedes learning. We introduce MeMo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. By design, MeMo offers transparency and the possibility of model editing, including forgetting texts. We experimented with the MeMo architecture, showing the memorization power of the one-layer and the multi-layer configurations.