mlsanityData to 14 Oct 2026

Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training

arXiv 2502.06902 ↗First posted 9 Feb 2025cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

We investigate in-context temporal biases in attention heads and transformer outputs. Using cognitive science methodologies, we analyze attention scores and outputs of the GPT-2 models of varying sizes. Across attention heads, we observe effects characteristic of human episodic memory, including temporal contiguity, primacy and recency. Transformer outputs demonstrate a tendency toward in-context serial recall. Importantly, this effect is eliminated after the ablation of the induction heads, which are the driving force behind the contiguity effect. Our findings offer insights into how transformers organize information temporally during in-context learning, shedding light on their similarities and differences with human memory and learning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 15 of its 41 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

15 of 41 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  8. 2022Training Language Models with Memory Augmentationcited by 113
  9. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  10. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  11. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  12. 2023Primacy Effect of ChatGPTcited by 27
3 more
  1. 2024Linking In-context Learning in Transformers to Human Episodic Memorycited by 15
  2. 2024Assessing Episodic Memory in LLMs with Sequence Order Recall Taskscited by 10
  3. 2022LaMemo: Language Modeling with Look-Ahead Memorycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025User-Assistant Bias in LLMscited by 3
  2. 2025Sequence-to-Sequence Models with Attention Mechanistically Map to the Architecture of Human Memory Searchcited by 1
  3. 2025Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Modelscited by 0
  4. 2026Temporal Dependencies in In-Context Learning: The Role of Induction Headscited by 0