mlsanityData to 14 Oct 2026

Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition

arXiv 2410.05603 ↗First posted 8 Oct 2024cs.LG cs.AI cs.CLcited by 14

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities. In this study, we explore a surprising phenomenon related to ICL: LLMs can perform multiple, computationally distinct ICL tasks simultaneously, during a single inference call, a capability we term "task superposition". We provide empirical evidence of this phenomenon across various LLM families and scales and show that this phenomenon emerges even if we train the model to in-context learn one task at a time. We offer theoretical explanations that this capability is well within the expressive power of transformers. We also explore how LLMs internally compose task vectors during superposition. Furthermore, we show that larger models can solve more ICL tasks in parallel, and better calibrate their output distribution. Our findings offer insights into the latent capabilities of LLMs, further substantiate the perspective of "LLMs as superposition of simulators", and raise questions about the mechanisms enabling simultaneous task execution.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 40 of its 46 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

40 of 46 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2022Editing Models with Task Arithmeticcited by 1,120
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2023Larger language models do in-context learning differentlycited by 394
  11. 2023In-Context Learning Creates Task Vectorscited by 312
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
28 more
  1. 2023Function Vectors in Large Language Modelscited by 308
  2. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  3. 2023Role-Play with Large Language Modelscited by 294
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  5. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  6. 2022Can language models learn from explanations in context?cited by 249
  7. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  8. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  9. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  10. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  11. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  12. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  13. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
  14. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  15. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  16. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  17. 2023In-Context Learning through the Bayesian Prismcited by 83
  18. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  19. 2019Superposition of many models into onecited by 64
  20. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  21. 2024Is Mamba Capable of In-Context Learning?cited by 45
  22. 2024Dual Operating Modes of In-Context Learningcited by 43
  23. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 29
  24. 2022DataMUX: Data Multiplexing for Neural Networkscited by 18
  25. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
  26. 2021Multiversal views on language modelscited by 6
  27. 2024Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Passcited by 4
  28. 2024Identifying and Analyzing Performance-Critical Tokens in Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  2. 2024Practical Considerations for Agentic LLM Systemscited by 9
  3. 2025Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitationscited by 9
  4. 2025Just-in-time and distributed task representations in language modelscited by 9
  5. 2025In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Flycited by 5
  6. 2026Regulatory gray areas of LLM Termscited by 2
  7. 2026Shared Lexical Task Representations Explain Behavioral Variability In LLMscited by 2
  8. 2025Exploring Task Performance with Interpretable Models via Sparse Auto-Encoderscited by 1
  9. 2026UsefulBench: Towards Decision-Useful Information as a Target for Information Retrievalcited by 1
  10. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  11. 2026TabPFN-MT: A Natively Multitask In-Context Learner for Tabular Datacited by 0
  12. 2026Distributed Sparse Interventions in Language Modelscited by 0
2 more
  1. 2026Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMscited by 0
  2. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  2. 2023Concept-aware Training Improves In-context Learning Ability of Language Models
  3. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
  4. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learning
  5. 2026Understanding Large Language Models
  6. 2025Rethinking the Unsolvable: When In-Context Search Meets Test-Time Scaling
  7. 2025Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors
  8. 2023TART: A plug-and-play Transformer module for task-agnostic reasoning