mlsanityData to 14 Oct 2026

When can transformers compositionally generalize in-context?

arXiv 2407.12275 ↗First posted 17 Jul 2024cs.LG cs.NEcited by 7

Not on the map. Why?

Abstract (arXiv)

Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during training. Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components? Here we study a modular multitask setting that allows us to precisely control compositional structure in the data generation process. We present evidence that transformers learning in-context struggle to generalize compositionally on this task despite being in principle expressive enough to do so. Compositional generalization becomes possible only when introducing a bottleneck that enforces an explicit separation between task inference and task execution.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 10 of its 13 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

10 of 13 references.

References · 10

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2016HyperNetworkscited by 1,027
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  4. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  5. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  6. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  7. 2022Is a Modular Architecture Enough?cited by 36
  8. 2022On the Compositional Generalization Gap of In-Context Learningcited by 24
  9. 2023Discovering modular solutions that generalize compositionallycited by 23
  10. 2024Attention as a Hypernetworkcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  2. 2025Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Surveycited by 5
  3. 2025Distinct Computations Emerge From Compositional Curricula in In-Context Learningcited by 5
  4. 2025Understanding the Staged Dynamics of Transformers in Learning Latent Structurecited by 1
  5. 2026Functorial Neural Architectures from Higher Inductive Typescited by 0
  6. 2026Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoningcited by 0
  7. 2026Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMscited by 0