# Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis

arXiv: https://arxiv.org/abs/2509.24164 · HTML: https://mlsanity.com/paper/2509.24164/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)
- [Language Models Implement Simple Word2Vec-style Vector Arithmetic](https://mlsanity.com/paper/2305.16130/) (2023)
- [Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2402.18154/) (2024)
- [What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation](https://mlsanity.com/paper/2404.07129/) (2024)
- [Overthinking the Truth: Understanding how Language Models Process False Demonstrations](https://mlsanity.com/paper/2307.09476/) (2023)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Which Attention Heads Matter for In-Context Learning?](https://mlsanity.com/paper/2502.14010/) (2025)
- [Identifying Semantic Induction Heads to Understand In-Context Learning](https://mlsanity.com/paper/2402.13055/) (2024)
- [Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning](https://mlsanity.com/paper/2407.07011/) (2024)
- [Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs](https://mlsanity.com/paper/2402.07321/) (2024)
- [Iteration Head: A Mechanistic Study of Chain-of-Thought](https://mlsanity.com/paper/2406.02128/) (2024)
- [Out-of-distribution generalization via composition: a lens through induction heads in Transformers](https://mlsanity.com/paper/2408.09503/) (2024)
- [Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution](https://mlsanity.com/paper/2410.00153/) (2024)
- [How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning](https://mlsanity.com/paper/2402.02872/) (2024)
- [The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models](https://mlsanity.com/paper/2502.08009/) (2025)
- [Revisiting In-context Learning Inference Circuit in Large Language Models](https://mlsanity.com/paper/2410.04468/) (2024)
- [Large Language Models Encode Semantics and Alignment in Linearly Separable Representations](https://mlsanity.com/paper/2507.09709/) (2025)
- [The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation](https://mlsanity.com/paper/2505.15807/) (2025)
- [Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning](https://mlsanity.com/paper/2505.18752/) (2025)
- [Token-based Decision Criteria Are Suboptimal in In-context Learning](https://mlsanity.com/paper/2406.16535/) (2024)
- [From Compression to Expression: A Layerwise Analysis of In-Context Learning](https://mlsanity.com/paper/2505.17322/) (2025)

## Built on by (most cited first, 2 of 2)

- [Task Context Shapes Component Roles in Language Models](https://mlsanity.com/paper/2610.13895/) (2026)
- [From Copying to Composition: How Language Models Develop In-Context Learning](https://mlsanity.com/paper/2610.19113/) (2026)

## Abstract (arXiv)

We investigate the mechanistic underpinnings of in-context learning (ICL) in large language models by reconciling two dominant perspectives: the component-level analysis of attention heads and the holistic decomposition of ICL into Task Recognition (TR) and Task Learning (TL). We propose a novel framework based on Task Subspace Logit Attribution (TSLA) to identify attention heads specialized in TR and TL, and demonstrate their distinct yet complementary roles. Through correlation analysis, ablation studies, and input perturbations, we show that the identified TR and TL heads independently and effectively capture the TR and TL components of ICL. Using steering experiments with geometric analysis of hidden states, we reveal that TR heads promote task recognition by aligning hidden states with the task subspace, while TL heads rotate hidden states within the subspace toward the correct label to facilitate prediction. We further show how previous findings on ICL mechanisms, including induction heads and task vectors, can be reconciled with our attention-head-level analysis of the TR-TL decomposition. Our framework thus provides a unified and interpretable account of how large language models execute ICL across diverse tasks and settings.