# On The Specialization of Neural Modules arXiv: https://arxiv.org/abs/2409.14981 ยท HTML: https://mlsanity.com/paper/2409.14981/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 55 references that are in the map. - First posted on arXiv 23 Sep 2024. ## Builds on (most cited first, 27 of 27) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016) - [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013) - [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015) - [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018) - [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019) - [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019) - [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018) - [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021) - [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017) - [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020) - [Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input](https://mlsanity.com/paper/1804.03984/) (2018) - [An analytic theory of generalization dynamics and transfer learning in deep linear networks](https://mlsanity.com/paper/1809.10374/) (2018) - [Measuring Compositionality in Representation Learning](https://mlsanity.com/paper/1902.07181/) (2019) - [Failures of Gradient-Based Deep Learning](https://mlsanity.com/paper/1703.07950/) (2017) - [Compositional Languages Emerge in a Neural Iterated Learning Model](https://mlsanity.com/paper/2002.01365/) (2020) - [CLOSURE: Assessing Systematic Generalization of CLEVR Models](https://mlsanity.com/paper/1912.05783/) (2019) - [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021) - [Explainable Neural Computation via Stack Neural Module Networks](https://mlsanity.com/paper/1807.08556/) (2018) - [Human few-shot learning of compositional instructions](https://mlsanity.com/paper/1901.04587/) (2019) - [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022) - [Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks](https://mlsanity.com/paper/2204.02892/) (2022) - [Is a Modular Architecture Enough?](https://mlsanity.com/paper/2206.02713/) (2022) - [Automatically Composing Representation Transformations as a Means for Generalization](https://mlsanity.com/paper/1807.04640/) (2018) - [Iterated learning for emergent systematicity in VQA](https://mlsanity.com/paper/2105.01119/) (2021) - [Emergent Language Generalization and Acquisition Speed are not tied to Compositionality](https://mlsanity.com/paper/2004.03420/) (2020) - [On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training](https://mlsanity.com/paper/1604.06915/) (2016) ## Built on by (most cited first, 14 of 14) - [A Complexity-Based Theory of Compositionality](https://mlsanity.com/paper/2410.14817/) (2024) - [Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)](https://mlsanity.com/paper/2502.21009/) (2025) - [Scaling can lead to compositional generalization](https://mlsanity.com/paper/2507.07207/) (2025) - [A Theory of Initialisation's Impact on Specialisation](https://mlsanity.com/paper/2503.02526/) (2025) - [Range, not Independence, Drives Modularity in Biologically Inspired Representations](https://mlsanity.com/paper/2410.06232/) (2024) - [Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks](https://mlsanity.com/paper/2503.06181/) (2025) - [Systematic Generalization in Language Models Scales with Information Entropy](https://mlsanity.com/paper/2505.13089/) (2025) - [Modularity is the Bedrock of Natural and Artificial Intelligence](https://mlsanity.com/paper/2602.18960/) (2026) - [Neural Modular Physics for Elastic Simulation](https://mlsanity.com/paper/2512.15083/) (2025) - [A Theoretical Analysis of Provable Compositional Generalization in Neural Networks: A Necessary and Sufficient Condition](https://mlsanity.com/paper/2505.02627/) (2025) - [Game-Theoretic Gradient Control for Robust Neural Network Training](https://mlsanity.com/paper/2507.19143/) (2025) - [A mathematical theory of balancing relational generalization and memorization](https://mlsanity.com/paper/2605.22972/) (2026) - [How can embedding models bind concepts?](https://mlsanity.com/paper/2605.31503/) (2026) - [EMAN: Optimization-Driven Capacity Growth through Path Emergence in Multi-Task Learning](https://mlsanity.com/paper/2608.16930/) (2026) ## Abstract (arXiv) A number of machine learning models have been proposed with the goal of achieving systematic generalization: the ability to reason about new situations by combining aspects of previous experiences. These models leverage compositional architectures which aim to learn specialized modules dedicated to structures in a task that can be composed to solve novel problems with similar structures. While the compositionality of these architectures is guaranteed by design, the modules specializing is not. Here we theoretically study the ability of network modules to specialize to useful structures in a dataset and achieve systematic generalization. To this end we introduce a minimal space of datasets motivated by practical systematic generalization benchmarks. From this space of datasets we present a mathematical definition of systematicity and study the learning dynamics of linear neural modules when solving components of the task. Our results shed light on the difficulty of module specialization, what is required for modules to successfully specialize, and the necessity of modular architectures to achieve systematicity. Finally, we confirm that the theoretical results in our tractable setting generalize to more complex datasets and non-linear architectures.