mlsanityData to 7 Oct 2026

Rethinking Associative Memory Mechanism in Induction Head

arXiv 2412.11459 ↗First posted 16 Dec 2024cs.CL cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Induction head mechanism is a part of the computational circuits for in-context learning (ICL) that enable large language models (LLMs) to adapt to new tasks without fine-tuning. Most existing work explains the training dynamics behind acquiring such a powerful mechanism. However, the model's ability to coordinate in-context information over long contexts and global knowledge acquired during pretraining remains poorly understood. This paper investigates how a two-layer transformer thoroughly captures in-context information and balances it with pretrained bigram knowledge in next token prediction, from the viewpoint of associative memory. We theoretically analyze the representation of weight matrices in attention layers and the resulting logits when a transformer is given prompts generated by a bigram model. In the experiments, we design specific prompts to evaluate whether the outputs of the trained transformer align with the theoretical results.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 63 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

36 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers5 Jun

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
23 more
  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  3. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  4. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  5. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  6. 2022Analyzing Transformers in Embedding Spacecited by 131
  7. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  8. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  9. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  10. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  11. 2023In-Context Convergence of Transformerscited by 112
  12. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  13. 2023Length Generalization in Arithmetic Transformerscited by 64
  14. 2019Encoding word order in complex embeddingscited by 53
  15. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 46
  16. 2023Learning Transformer Programscited by 40
  17. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  18. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  19. 2023Scaling Laws for Associative Memoriescited by 30
  20. 2022The Curious Case of Absolute Position Embeddingscited by 17
  21. 2024Memory Mosaicscited by 16
  22. 2024Learning Associative Memories with Gradient Descentcited by 14
  23. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Linear Predictability of Attention Heads in Large Language Models
  2. 2025Pretraining with hierarchical memories: separating long-tail and common knowledge
  3. 2026Mixture of Chapters: Scaling Learnt Memory in Transformers
  4. 2026Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
  5. 2025Revisiting associative recall in modern recurrent models
  6. 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
  7. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learning
  8. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability