mlsanityData to 7 Oct 2026

In-Context Learning Creates Task Vectors

arXiv 2310.15916 ↗First posted 24 Oct 2023cs.CLcited by 302

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set S to find a best-fitting function f(x) in some hypothesis class. Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single "task vector" calculated from the training set. Thus, ICL can be seen as compressing S into a single task vector boldsymbolθ(S) and then using this task vector to modulate the transformer to produce the output. We support the above claim via comprehensive experiments across a range of models and tasks.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). Builds on 10 of its 21 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

10 of 21 references.

References · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  10. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 91; 2025: 102; 2026: 100.

302 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  4. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  5. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  6. 2024Many-Shot In-Context Learningcited by 203
  7. 2024How to use and interpret activation patchingcited by 191
  8. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  9. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  10. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  11. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  12. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
38 more
  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  2. 2024Universal Neurons in GPT2 Language Modelscited by 100
  3. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  4. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  5. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  6. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  7. 2024Can large language models explore in-context?cited by 75
  8. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  9. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 62
  10. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  11. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  12. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  13. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  14. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  15. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  16. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  17. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  18. 2024ICLR: In-Context Learning of Representationscited by 49
  19. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  20. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  21. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 46
  22. 2024Is Mamba Capable of In-Context Learning?cited by 45
  23. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  24. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  25. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  26. 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 39
  27. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  28. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 37
  29. 2024Multi-property Steering of Large Language Models with Dynamic Activation Compositioncited by 35
  30. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  31. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  32. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  33. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  34. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  35. 2024LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling lawcited by 32
  36. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  37. 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
  38. 2024Linear Transformers are Versatile In-Context Learnerscited by 28

50 most cited of 302 All 302 (JSON)