mlsanityData to 14 Oct 2026

Context-Gated Associative Retrieval: From Theory to Transformers

arXiv 2605.10970 ↗First posted 8 May 2026cond-mat.dis-nn cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Hopfield networks and their generalizations have established deep connections among biological associative memories, statistical physics, and transformers. Yet most models treat retrieval as a fixed query-to-memory mapping, ignoring the role of external context in recall. In this work, we propose a two-stage associative memory architecture, wherein a context-gate subcircuit reshapes the retrieval energy landscape before and during recall. We show theoretically that context gating increases inter-memory separation while inducing sparsity, translating into exponential improvements in retrieval. Crucially, we prove that the system admits a unique self-consistent fixed point, revealing that the resulting retrieval state is driven by both a direct contextual bias and a second-order retrieval-gate feedback loop. We then bridge this theory to transformers; specifically, we evaluate a first-order approximation on Llama-3, confirming that in-context learning acts as context-gated retrieval. Native dynamics mirror our theory: context localizes a memory subspace, enabling the zero-shot query to cleanly discriminate. Ultimately, this framework provides a mechanistic link between associative memory theory and LLM phenomenology.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

18 of 35 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2020Hopfield Networks is All You Needcited by 475
  6. 2023Function Vectors in Large Language Modelscited by 308
  7. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  8. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  9. 2023On Sparse Modern Hopfield Modelcited by 42
  10. 2025Memorization to Generalization: Emergence of Diffusion Models from Associative Memorycited by 36
  11. 2021Hierarchical Associative Memorycited by 34
  12. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
6 more
  1. 2025Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learningcited by 10
  2. 2024Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrievalcited by 6
  3. 2026A Biologically Plausible Dense Associative Memory with Exponential Capacitycited by 6
  4. 2024Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networkscited by 5
  5. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  6. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understandingcited by 2

Built on by

Not cited yet.