# Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning

arXiv: https://arxiv.org/abs/2406.06262 · HTML: https://mlsanity.com/paper/2406.06262/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 67 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [Weight Agnostic Neural Networks](https://mlsanity.com/paper/1906.04358/) (2019)
- [Superposition of many models into one](https://mlsanity.com/paper/1902.05522/) (2019)
- [Can recurrent neural networks warp time?](https://mlsanity.com/paper/1804.11188/) (2018)
- [Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks](https://mlsanity.com/paper/2005.11633/) (2020)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [A Review of Modularization Techniques in Artificial Neural Networks](https://mlsanity.com/paper/1904.12770/) (2019)
- [Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programs](https://mlsanity.com/paper/2307.08197/) (2023)
- [Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation](https://mlsanity.com/paper/2306.12700/) (2023)
- [Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks](https://mlsanity.com/paper/2309.12927/) (2023)
- [Harmonizing Program Induction with Rate-Distortion Theory](https://mlsanity.com/paper/2405.05294/) (2024)
- [Spatial and temporal correlations in neural networks with structured connectivity](https://mlsanity.com/paper/2207.07930/) (2022)
- [Emergence of slow-switching assemblies in structured neuronal networks](https://mlsanity.com/paper/1502.05656/) (2015)
- [Tailored ensembles of neural networks optimize sensitivity to stimulus statistics](https://mlsanity.com/paper/1905.10401/) (2019)
- [When to be critical? Performance and evolvability in different regimes of neural Ising agents](https://mlsanity.com/paper/2303.16195/) (2023)

## Built on by (most cited first, 2 of 2)

- [Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisation](https://mlsanity.com/paper/2512.13707/) (2025)
- [Distilling a Modular Reservoir Through a Genomic Bottleneck](https://mlsanity.com/paper/2606.28380/) (2026)

## Abstract (arXiv)

Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth curriculum. We find that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics, including training time, generalizability, and robustness to some perturbations. We further examine how different aspects of a modular network's connectivity contribute to its computational capability. We then demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed and only the connections between modules are trained. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales, and help build more scalable and compressible artificial networks.