# Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics

arXiv: https://arxiv.org/abs/2409.09626 · HTML: https://mlsanity.com/paper/2409.09626/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Characterizing Structural Regularities of Labeled Data in Overparameterized Models](https://mlsanity.com/paper/2002.03206/) (2020)
- [Measuring Compositionality in Representation Learning](https://mlsanity.com/paper/1902.07181/) (2019)
- [Compositional Languages Emerge in a Neural Iterated Learning Model](https://mlsanity.com/paper/2002.01365/) (2020)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Improving Compositional Generalization with Latent Structure and Data Augmentation](https://mlsanity.com/paper/2112.07610/) (2021)
- [The Local Elasticity of Neural Networks](https://mlsanity.com/paper/1910.06943/) (2019)
- [Better Supervisory Signals by Observing Learning Paths](https://mlsanity.com/paper/2203.02485/) (2022)
- [How to prepare your task head for finetuning](https://mlsanity.com/paper/2302.05779/) (2023)
- [Compositional Networks Enable Systematic Generalization for Grounded Language Understanding](https://mlsanity.com/paper/2008.02742/) (2020)
- [Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings](https://mlsanity.com/paper/2310.18777/) (2023)
- [lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning](https://mlsanity.com/paper/2401.08808/) (2024)

## Built on by (most cited first, 5 of 5)

- [A Complexity-Based Theory of Compositionality](https://mlsanity.com/paper/2410.14817/) (2024)
- [Does Data Scaling Lead to Visual Compositional Generalization?](https://mlsanity.com/paper/2507.07102/) (2025)
- [Scaling can lead to compositional generalization](https://mlsanity.com/paper/2507.07207/) (2025)
- [How can embedding models bind concepts?](https://mlsanity.com/paper/2605.31503/) (2026)
- [Understanding Generalization Requires Universal Induction](https://mlsanity.com/paper/2609.34458/) (2026)

## Abstract (arXiv)

Obtaining compositional mappings is important for the model to generalize well compositionally. To better understand when and how to encourage the model to learn such mappings, we study their uniqueness through different perspectives. Specifically, we first show that the compositional mappings are the simplest bijections through the lens of coding length (i.e., an upper bound of their Kolmogorov complexity). This property explains why models having such mappings can generalize well. We further show that the simplicity bias is usually an intrinsic property of neural network training via gradient descent. That partially explains why some models spontaneously generalize well when they are trained appropriately.