# Dynamics of specialization in neural modules under resource constraints

arXiv: https://arxiv.org/abs/2106.02626 · HTML: https://mlsanity.com/paper/2106.02626/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 73 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 12 of 12)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Modular Networks: Learning to Decompose Neural Computation](https://mlsanity.com/paper/1811.05249/) (2018)
- [Clusterability in Neural Networks](https://mlsanity.com/paper/2103.03386/) (2021)
- [The Dynamics of Functional Brain Networks: Integrated Network States during Cognitive Function](https://mlsanity.com/paper/1511.02976/) (2015)
- [Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations](https://mlsanity.com/paper/2004.01254/) (2020)
- [Interpreting Layered Neural Networks via Hierarchical Modular Representation](https://mlsanity.com/paper/1810.01588/) (2018)
- [Generalisation of structural knowledge in the hippocampal-entorhinal system](https://mlsanity.com/paper/1805.09042/) (2018)
- [Community detection in network neuroscience](https://mlsanity.com/paper/2011.06723/) (2020)

## Built on by (most cited first, 6 of 6)

- [Modularity is the Bedrock of Natural and Artificial Intelligence](https://mlsanity.com/paper/2602.18960/) (2026)
- [What You Can't See Is What You Learn: Slot-Selective Evidence Masking Favors Compositional Generalization in Shared-Genome Language-Model Societies](https://mlsanity.com/paper/2608.20054/) (2026)
- [When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry](https://mlsanity.com/paper/2604.27656/) (2026)
- [Unifying Dynamical Systems and Graph Theory to Mechanistically Understand Computation in Neural Networks](https://mlsanity.com/paper/2605.03598/) (2026)
- [Dimensionality Controls When Modularity Helps in Continual Learning](https://mlsanity.com/paper/2606.17889/) (2026)
- [SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization](https://mlsanity.com/paper/2608.04084/) (2026)

## Abstract (arXiv)

It has long been believed that the brain is highly modular both in terms of structure and function, although recent evidence has led some to question the extent of both types of modularity. We used artificial neural networks to test the hypothesis that structural modularity is sufficient to guarantee functional specialization, and find that in general, this doesn't necessarily hold. We then systematically tested which features of the environment and network do lead to the emergence of specialization. We used a simple toy environment, task and network, allowing us precise control, and show that in this setup, several distinct measures of specialization give qualitatively similar results. We further find that in this setup (1) specialization can only emerge in environments where features of that environment are meaningfully separable, (2) specialization preferentially emerges when the network is strongly resource-constrained, and (3) these findings are qualitatively similar across the different variations of network architectures that we tested, but that the quantitative relationships depend on the precise architecture. Finally, we show that functional specialization varies dynamically across time, and demonstrate that these dynamics depend on both the timing and bandwidth of information flow in the network. We conclude that a static notion of specialization, based on structural modularity, is likely too simple a framework for understanding intelligence in situations of real-world complexity, from biology to brain-inspired neuromorphic systems. We propose that thoroughly stress testing candidate definitions of functional modularity in simplified scenarios before extending to more complex data, network models and electrophysiological recordings is likely to be a fruitful approach.