# Linking In-context Learning in Transformers to Human Episodic Memory

arXiv: https://arxiv.org/abs/2405.14992 · HTML: https://mlsanity.com/paper/2405.14992/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 64 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [The mechanistic basis of data dependence and abrupt learning in an in-context classification task](https://mlsanity.com/paper/2312.03002/) (2023)
- [What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation](https://mlsanity.com/paper/2404.07129/) (2024)
- [Copy Suppression: Comprehensively Understanding an Attention Head](https://mlsanity.com/paper/2310.04625/) (2023)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale](https://mlsanity.com/paper/2212.09095/) (2022)
- [Relating transformers to models and neural representations of the hippocampal formation](https://mlsanity.com/paper/2112.04035/) (2021)
- [Emergence of grid-like representations by training recurrent neural networks to perform spatial localization](https://mlsanity.com/paper/1803.07770/) (2018)
- [Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability](https://mlsanity.com/paper/2310.08049/) (2023)

## Built on by (most cited first, 15 of 15)

- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Human-inspired Episodic Memory for Infinite Context LLMs](https://mlsanity.com/paper/2407.09450/) (2024)
- [Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks](https://mlsanity.com/paper/2410.08133/) (2024)
- [Position: Modular Memory is the Key to Continual Learning Agents](https://mlsanity.com/paper/2603.01761/) (2026)
- [Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training](https://mlsanity.com/paper/2502.06902/) (2025)
- [Schema for In-Context Learning](https://mlsanity.com/paper/2510.13905/) (2025)
- [Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture](https://mlsanity.com/paper/2412.15113/) (2024)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Sequence-to-Sequence Models with Attention Mechanistically Map to the Architecture of Human Memory Search](https://mlsanity.com/paper/2506.17424/) (2025)
- [Large Language Models as Model Organisms for Human Associative Learning](https://mlsanity.com/paper/2510.21408/) (2025)
- [Medical Interpretability and Knowledge Maps of Large Language Models](https://mlsanity.com/paper/2510.11390/) (2025)
- [Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models](https://mlsanity.com/paper/2510.22752/) (2025)
- [Temporal Dependencies in In-Context Learning: The Role of Induction Heads](https://mlsanity.com/paper/2604.01094/) (2026)
- [Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models](https://mlsanity.com/paper/2607.22575/) (2026)
- [Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognition](https://mlsanity.com/paper/2607.26179/) (2026)

## Abstract (arXiv)

Understanding connections between artificial and biological intelligent systems can reveal fundamental principles of general intelligence. While many artificial intelligence models have a neuroscience counterpart, such connections are largely missing in Transformer models and the self-attention mechanism. Here, we examine the relationship between interacting attention heads and human episodic memory. We focus on induction heads, which contribute to in-context learning in Transformer-based large language models (LLMs). We demonstrate that induction heads are behaviorally, functionally, and mechanistically similar to the contextual maintenance and retrieval (CMR) model of human episodic memory. Our analyses of LLMs pre-trained on extensive text data show that CMR-like heads often emerge in the intermediate and late layers, qualitatively mirroring human memory biases. The ablation of CMR-like heads suggests their causal role in in-context learning. Our findings uncover a parallel between the computational mechanisms of LLMs and human memory, offering valuable insights into both research fields.