# Task Vectors in In-Context Learning: Emergence, Formation, and Benefit

arXiv: https://arxiv.org/abs/2501.09240 · HTML: https://mlsanity.com/paper/2501.09240/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2025.

## Built on by (most cited first, 20 of 20)

- [Learning without training: The implicit dynamics of in-context learning](https://mlsanity.com/paper/2507.16003/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Text Generation Beyond Discrete Token Sampling](https://mlsanity.com/paper/2505.14827/) (2025)
- [Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs](https://mlsanity.com/paper/2505.14530/) (2025)
- [Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations](https://mlsanity.com/paper/2506.09048/) (2025)
- [Just-in-time and distributed task representations in language models](https://mlsanity.com/paper/2509.04466/) (2025)
- [Scaling can lead to compositional generalization](https://mlsanity.com/paper/2507.07207/) (2025)
- [Adaptive Task Vectors for Large Language Models](https://mlsanity.com/paper/2506.03426/) (2025)
- [Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning](https://mlsanity.com/paper/2511.08246/) (2025)
- [Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight](https://mlsanity.com/paper/2509.24169/) (2025)
- [Transmuting prompts into weights](https://mlsanity.com/paper/2510.08734/) (2025)
- [Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge](https://mlsanity.com/paper/2603.20969/) (2026)
- [In-Context Learning Operates as Concept Subspace Learning](https://mlsanity.com/paper/2605.18830/) (2026)
- [Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective](https://mlsanity.com/paper/2506.16288/) (2025)
- [Look the Other Way: Designing 'Positive' Molecules with Negative Data via Task Arithmetic](https://mlsanity.com/paper/2507.17876/) (2025)
- [Beyond Plain Demos: A Demo-centric Anchoring Paradigm for In-Context Learning in Alzheimer's Disease Detection](https://mlsanity.com/paper/2511.06826/) (2025)
- [Reconciling In-Context and In-Weight Learning via Dual Representation Space Encoding](https://mlsanity.com/paper/2603.13459/) (2026)
- [DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning](https://mlsanity.com/paper/2604.11129/) (2026)
- [Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders](https://mlsanity.com/paper/2605.01372/) (2026)
- [Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning](https://mlsanity.com/paper/2605.20730/) (2026)

## Abstract (arXiv)

In-context learning is a remarkable capability of transformers, referring to their ability to adapt to specific tasks based on a short history or context. Previous research has found that task-specific information is locally encoded within models, though their emergence and functionality remain unclear due to opaque pre-training processes. In this work, we investigate the formation of task vectors in a controlled setting, using models trained from scratch on synthetic datasets. Our findings confirm that task vectors naturally emerge under certain conditions, but the tasks may be relatively weakly and/or non-locally encoded within the model. To promote strong task vectors encoded at a prescribed location within the model, we propose an auxiliary training mechanism based on a task vector prompting loss (TVP-loss). This method eliminates the need to search for task-correlated encodings within the trained model and demonstrably improves robustness and generalization.