mlsanityData to 7 Oct 2026

In-Context Exemplars as Clues to Retrieving from Large Associative Memory

arXiv 2311.03498 ↗First posted 6 Nov 2023cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recently, large language models (LLMs) have made remarkable progress in natural language processing. The most representative ability of LLMs is in-context learning (ICL), which enables LLMs to learn patterns from in-context exemplars without training. The performance of ICL greatly depends on the exemplars used. However, how to choose exemplars remains unclear due to the lack of understanding of how in-context learning works. In this paper, we present a novel perspective on ICL by conceptualizing it as contextual retrieval from a model of associative memory. We establish a theoretical framework of ICL based on Hopfield Networks. Based on our framework, we look into how in-context exemplars influence the performance of ICL and propose more efficient active exemplar selection. Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval, with potential implications for advancing the understanding of LLMs.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

37 of 59 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
25 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2022Transformers learn in-context by gradient descentcited by 714
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  5. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  6. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  7. 2020Hopfield Networks is All You Needcited by 461
  8. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  9. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  10. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  11. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  12. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  13. 2022A Review on Language Models as Knowledge Basescited by 156
  14. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  15. 2023Compositional Exemplars for In-context Learningcited by 143
  16. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 116
  17. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  18. 2022A Neural Corpus Indexer for Document Retrievalcited by 107
  19. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  20. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  21. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 52
  22. 2023On Sparse Modern Hopfield Modelcited by 41
  23. 2021Attention Approximates Sparse Distributed Memorycited by 29
  24. 2017Frustratingly Short Attention Spans in Neural Language Modelingcited by 18
  25. 2016Can Active Memory Replace Attention?cited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 8.

13 papers.

Sort:

By area:

  1. 2025Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositionscited by 15
  2. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  3. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  4. 2024Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategiescited by 10
  5. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-contextcited by 10
  6. 2023Large Language Models are In-context Teachers for Knowledge Reasoningcited by 4
  7. 2025ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Cipherscited by 4
  8. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  9. 2025Context Tuning for In-Context Optimizationcited by 2
  10. 2025Swallowing the Poison Pills: Insights from Vulnerability Disparity Among LLMscited by 1
  11. 2025Large Language Models as Model Organisms for Human Associative Learningcited by 1
  12. 2025What Neuroscience Can Teach AI About Learning in Continuously Changing Environmentscited by 0
1 more
  1. 2025MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Promptscited by 0