mlsanityData to 14 Oct 2026

The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation

arXiv 2505.15807 ↗First posted 21 May 2025cs.CL cs.IR cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models are able to exploit in-context learning to access external knowledge beyond their training data through retrieval-augmentation. While promising, its inner workings remain unclear. In this work, we shed light on the mechanism of in-context retrieval augmentation for question answering by viewing a prompt as a composition of informational components. We propose an attribution-based method to identify specialized attention heads, revealing in-context heads that comprehend instructions and retrieve relevant contextual information, and parametric heads that store entities' relational knowledge. To better understand their roles, we extract function vectors and modify their attention weights to show how they can influence the answer generation process. Finally, we leverage the gained insights to trace the sources of knowledge used during inference, paving the way towards more safe and transparent language models.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 77 of its 89 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

77 of 89 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  11. 2019Attention is not Explanationcited by 929
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  13. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  14. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  15. 2022Transformers learn in-context by gradient descentcited by 732
  16. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  17. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  18. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  19. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  20. 2019Attention is not not Explanationcited by 636
  21. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  22. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  23. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  24. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  25. 2021Causal Abstractions of Neural Networkscited by 422
  26. 2019Is Attention Interpretable?cited by 389
  27. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  28. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  29. 2023In-Context Learning Creates Task Vectorscited by 312
  30. 2023Function Vectors in Large Language Modelscited by 308
  31. 2019Revealing the Dark Secrets of BERTcited by 301
  32. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  33. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  34. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  35. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  36. 2019Understanding and Improving Layer Normalizationcited by 217
  37. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  38. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 184

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2026RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provablycited by 8
  2. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  3. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
  4. 2025Interpreting Language Models Through Concept Descriptions: A Surveycited by 3
  5. 2025Atlas-Alignment: Making Interpretability Transferable Across Language Modelscited by 3
  6. 2025Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysiscited by 2
  7. 2026Fast & Faithful Function Vectorscited by 0
  8. 2026Mechanistic Attention Guidance for Agent Memory Refinementcited by 0
  9. 2026ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformerscited by 0
  10. 2026The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstainingcited by 0
  11. 2026Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoningcited by 0