mlsanityData to 14 Oct 2026

Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations

arXiv 2506.09048 ↗First posted 10 Jun 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Task vectors offer a compelling mechanism for accelerating inference in in-context learning (ICL) by distilling task-specific information into a single, reusable representation. Despite their empirical success, the underlying principles governing their emergence and functionality remain unclear. This work proposes the Linear Combination Conjecture, positing that task vectors act as single in-context demonstrations formed through linear combinations of the original ones. We provide both theoretical and empirical support for this conjecture. First, we show that task vectors naturally emerge in linear transformers trained on triplet-formatted prompts through loss landscape analysis. Next, we predict the failure of task vectors on representing high-rank mappings and confirm this on practical LLMs. Our findings are further validated through saliency analyses and parameter visualization, suggesting an enhancement of task vectors by injecting multiple ones into few-shot prompts. Together, our results advance the understanding of task vectors and shed light on the mechanisms underlying ICL in transformer-based models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

23 of 26 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  6. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  10. 2023Function Vectors in Large Language Modelscited by 308
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  12. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
11 more
  1. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  2. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  3. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  4. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  5. 2024Finding Visual Task Vectorscited by 24
  6. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  7. 2025Task Vectors in In-Context Learning: Emergence, Formation, and Benefitcited by 20
  8. 2024In-Context Learning State Vector with Inner and Momentum Optimizationcited by 15
  9. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 14
  10. 2024Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddingscited by 8
  11. 2024Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Just-in-time and distributed task representations in language modelscited by 9
  2. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
  3. 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 3
  4. 2026In-Context Learning Operates as Concept Subspace Learningcited by 3
  5. 2026Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scalecited by 2
  6. 2026Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learningcited by 0
  7. 2026Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learningcited by 0
  8. 2026Capturing In-Context Learning Dynamics with Task Operatorscited by 0
  9. 2026Label Collapse in Diffusion Language Model In-Context Learningcited by 0