# Discovering modular solutions that generalize compositionally

arXiv: https://arxiv.org/abs/2312.15001 · HTML: https://mlsanity.com/paper/2312.15001/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2023.

## Built on by (most cited first, 23 of 23)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [On Provable Length and Compositional Generalization](https://mlsanity.com/paper/2402.04875/) (2024)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [Attention as a Hypernetwork](https://mlsanity.com/paper/2406.05816/) (2024)
- [Compositional Risk Minimization](https://mlsanity.com/paper/2410.06303/) (2024)
- [A Complexity-Based Theory of Compositionality](https://mlsanity.com/paper/2410.14817/) (2024)
- [G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration](https://mlsanity.com/paper/2506.09272/) (2025)
- [When can transformers compositionally generalize in-context?](https://mlsanity.com/paper/2407.12275/) (2024)
- [Scaling can lead to compositional generalization](https://mlsanity.com/paper/2507.07207/) (2025)
- [Range, not Independence, Drives Modularity in Biologically Inspired Representations](https://mlsanity.com/paper/2410.06232/) (2024)
- [Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attention](https://mlsanity.com/paper/2603.14483/) (2026)
- [Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity](https://mlsanity.com/paper/2410.13964/) (2024)
- [Flexible task abstractions emerge in linear networks with fast and bounded units](https://mlsanity.com/paper/2411.03840/) (2024)
- [Composing Linear Layers from Irreducibles](https://mlsanity.com/paper/2507.11688/) (2025)
- [An Alternative Trajectory for Generative AI](https://mlsanity.com/paper/2603.14147/) (2026)
- [Compositional Models for Estimating Causal Effects](https://mlsanity.com/paper/2406.17714/) (2024)
- [A Theoretical Analysis of Provable Compositional Generalization in Neural Networks: A Necessary and Sufficient Condition](https://mlsanity.com/paper/2505.02627/) (2025)
- [Integrative neurocybernetic modeling in the era of large-scale neuroscience](https://mlsanity.com/paper/2604.23903/) (2026)
- [A mathematical theory of balancing relational generalization and memorization](https://mlsanity.com/paper/2605.22972/) (2026)
- [The Limits of Binding in Dual Encoders](https://mlsanity.com/paper/2608.15971/) (2026)
- [EMAN: Optimization-Driven Capacity Growth through Path Emergence in Multi-Task Learning](https://mlsanity.com/paper/2608.16930/) (2026)
- [A Spectral Theory of Compositional Learning](https://mlsanity.com/paper/2609.33708/) (2026)
- [Knowledge transfer between simple discrete worlds](https://mlsanity.com/paper/2610.15081/) (2026)

## Abstract (arXiv)

Many complex tasks can be decomposed into simpler, independent parts. Discovering such underlying compositional structure has the potential to enable compositional generalization. Despite progress, our most powerful systems struggle to compose flexibly. It therefore seems natural to make models more modular to help capture the compositional nature of many tasks. However, it is unclear under which circumstances modular systems can discover hidden compositional structure. To shed light on this question, we study a teacher-student setting with a modular teacher where we have full control over the composition of ground truth modules. This allows us to relate the problem of compositional generalization to that of identification of the underlying modules. In particular we study modularity in hypernetworks representing a general class of multiplicative interactions. We show theoretically that identification up to linear transformation purely from demonstrations is possible without having to learn an exponential number of module combinations. We further demonstrate empirically that under the theoretically identified conditions, meta-learning from finite data can discover modular policies that generalize compositionally in a number of complex environments.