mlsanityData to 7 Oct 2026

Just-in-time and distributed task representations in language models

arXiv 2509.04466 ↗First posted 28 Aug 2025cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Many of language models' impressive capabilities originate from their in-context learning: based on instructions or examples, they can infer and perform new tasks without weight updates. In this work, we investigate when representations for new tasks are formed in language models, and how these representations change over the course of context. We study two different task representations: those that are ''transferrable'' -- vector representations that can transfer task contexts to another model instance, even without the full prompt -- and simpler representations of high-level task categories. We show that transferrable task representations evolve in non-monotonic and sporadic ways, while task identity representations persist throughout the context. Specifically, transferrable task representations exhibit a two-fold locality. They successfully condense evidence when more examples are provided in the context. But this evidence accrual process exhibits strong temporal locality along the sequence dimension, coming online only at certain tokens -- despite task identity being reliably decodable throughout the context. In some cases, transferrable task representations also show semantic locality, capturing a small task ''scope'' such as an independent subtask. Language models thus represent new tasks on the fly through both an inert, sustained sensitivity to the task and an active, just-in-time representation to support inference.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 28 Aug 2025.

Builds on

28 of 31 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Gemma 3 Technical Reportcited by 1,686
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  10. 2023In-Context Learning Creates Task Vectorscited by 302
  11. 2023Function Vectors in Large Language Modelscited by 292
  12. 2024Many-Shot In-Context Learningcited by 203
16 more
  1. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  2. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  3. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  4. 2024The broader spectrum of in-context learningcited by 33
  5. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  6. 2025Task Vectors in In-Context Learning: Emergence, Formation, and Benefitcited by 20
  7. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  8. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
  9. 2024Vision-Language Models Create Cross-Modal Task Representationscited by 14
  10. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  11. 2024Implicit In-context Learningcited by 11
  12. 2025Learning Task Representations from In-Context Learningcited by 11
  13. 2025Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2Bcited by 10
  14. 2025Do different prompting methods yield a common task representation in language models?cited by 9
  15. 2025Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitationscited by 8
  16. 2025One Task Vector is not Enough: A Large-Scale Study for In-Context Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 9
  2. 2026Linear representations in language models can change dramatically over a conversationcited by 7
  3. 2026Context Structure Reshapes the Representational Geometry of Language Modelscited by 6
  4. 2026Convergent World Representations and Divergent Taskscited by 1
  5. 2026Patches of Nonlinearity: Instruction Vectors in Large Language Modelscited by 1
  6. 2026Distributed Sparse Interventions in Language Modelscited by 0
  7. 2026One mechanism for many mental spaces: a shared router over a value slot in language modelscited by 0