mlsanityData to 7 Oct 2026

Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers

arXiv 2501.08537 ↗First posted 15 Jan 2025cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Transformers have demonstrated impressive capabilities across various tasks, yet their performance on compositional problems remains a subject of debate. In this study, we investigate the internal mechanisms underlying Transformers' behavior in compositional tasks. We find that complexity control strategies significantly influence whether the model learns primitive-level rules that generalize out-of-distribution (reasoning-based solutions) or relies solely on memorized mappings (memory-based solutions). By applying masking strategies to the model's information circuits and employing multiple complexity metrics, we reveal distinct internal working mechanisms associated with different solution types. Further analysis reveals that reasoning-based solutions exhibit a lower complexity bias, which aligns with the well-studied neuron condensation phenomenon. This lower complexity bias is hypothesized to be the key factor enabling these solutions to learn reasoning rules. We validate these conclusions across multiple real-world datasets, including image generation and natural language processing tasks, confirming the broad applicability of our findings.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 56 of its 82 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

56 of 82 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2017A Closer Look at Memorization in Deep Networkscited by 936
  10. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
38 more
  1. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  2. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  3. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  4. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  6. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  7. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  8. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  9. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  10. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  11. 2018Training behavior of deep neural network in frequency domaincited by 174
  12. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  13. 2022Faithful Reasoning Using Large Language Modelscited by 130
  14. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  15. 2022A very preliminary analysis of DALL-E 2cited by 82
  16. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  17. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  18. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  19. 2023Mimetic Initialization of Self-Attention Layerscited by 52
  20. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  21. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  22. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  23. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  24. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  25. 2020Assessing Phrasal Representation and Composition in Transformerscited by 41
  26. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  27. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  28. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  29. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  30. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  31. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  32. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  33. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  34. 2024GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understandingcited by 18
  35. 2023Loss Spike in Training Neural Networkscited by 15
  36. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  37. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  38. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  3. 2025An overview of condensation phenomenon in deep learningcited by 13
  4. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  5. 2026Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Taskscited by 10
  6. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 9
  7. 2025An explainable transformer circuit for compositional generalizationcited by 8
  8. 2026Weight Decay Improves Language Model Plasticitycited by 4
  9. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  10. 2025Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridgecited by 3
  11. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  12. 2025Frequency-Aware Ensemble Learning for BraTS 2025 Pediatric Brain Tumor Segmentationcited by 1
7 more
  1. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Modelscited by 1
  2. 2026Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorizecited by 1
  3. 2026Towards Compositional Generalization in LLMs for Smart Contract Security: A Case Study on Reentrancy Vulnerabilitiescited by 0
  4. 2026The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-trainingcited by 0
  5. 2026Small Initialization Matters for Large Language Modelscited by 0
  6. 2026Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learningcited by 0
  7. 2026Complexity-Guided Component-wise Initialization for Language Model Pretrainingcited by 0