mlsanityData to 14 Oct 2026

Distinct Computations Emerge From Compositional Curricula in In-Context Learning

arXiv 2506.13253 ↗First posted 16 Jun 2025cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) research often considers learning a function in-context through a uniform sample of input-output pairs. Here, we investigate how presenting a compositional subtask curriculum in context may alter the computations a transformer learns. We design a compositional algorithmic task based on the modular exponential-a double exponential task composed of two single exponential subtasks and train transformer models to learn the task in-context. We compare (a) models trained using an in-context curriculum consisting of single exponential subtasks and, (b) models trained directly on the double exponential task without such a curriculum. We show that models trained with a subtask curriculum can perform zero-shot inference on unseen compositional tasks and are more robust given the same context length. We study how the task and subtasks are represented across the two training regimes. We find that the models employ diverse strategies modulated by the specific curriculum design.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

45 of 65 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2016Matching Networks for One Shot Learningcited by 2,255
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2019Invariant Risk Minimizationcited by 1,854
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
33 more
  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  3. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  4. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  6. 2016Learning to reinforcement learncited by 377
  7. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  8. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  9. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  10. 2023In-Context Learning Creates Task Vectorscited by 312
  11. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  12. 2023Function Vectors in Large Language Modelscited by 308
  13. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  14. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  15. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  16. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  17. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  18. 2020When Do Curricula Work?cited by 106
  19. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  20. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  21. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  22. 2023On the Foundations of Shortcut Learningcited by 65
  23. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 64
  24. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  25. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  26. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  27. 2024The broader spectrum of in-context learningcited by 35
  28. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 34
  29. 2020Universal linguistic inductive biases via meta-learningcited by 29
  30. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  31. 2024Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learningcited by 16
  32. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 16
  33. 2024When can transformers compositionally generalize in-context?cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
  2. 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 8
  3. 2026Context Structure Reshapes the Representational Geometry of Language Modelscited by 8
  4. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 1
  5. 2026Pretraining Curricula Enable Selective Fine-tuningcited by 0