# Dimensionality Controls When Modularity Helps in Continual Learning

arXiv: https://arxiv.org/abs/2606.17889 · HTML: https://mlsanity.com/paper/2606.17889/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Disentangling and Mitigating the Impact of Task Similarity for Continual Learning](https://mlsanity.com/paper/2405.20236/) (2024)
- [How connectivity structure shapes rich and lazy learning in neural circuits](https://mlsanity.com/paper/2310.08513/) (2023)
- [Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective](https://mlsanity.com/paper/2506.03951/) (2025)
- [Neural population geometry and optimal coding of tasks with shared latent structure](https://mlsanity.com/paper/2402.16770/) (2024)
- [Dynamics of specialization in neural modules under resource constraints](https://mlsanity.com/paper/2106.02626/) (2021)
- [Emergence and reconfiguration of modular structure for synaptic neural networks during continual familiarity detection](https://mlsanity.com/paper/2311.05862/) (2023)
- [Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems](https://mlsanity.com/paper/2411.15234/) (2024)
- [Brain Modularity Mediates the Relation between Task Complexity and Performance](https://mlsanity.com/paper/1711.09133/) (2017)
- [Modularity is the Bedrock of Natural and Artificial Intelligence](https://mlsanity.com/paper/2602.18960/) (2026)

## Abstract (arXiv)

Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference. We study how modular architecture, task similarity, and representational dimensionality jointly shape compositional continual learning in a sequential A-B-A paradigm, comparing a task-partitioned recurrent network to a single-network baseline while inducing high- and low-dimensional regimes via weight-scale manipulations. In a high-dimensional "lazy" regime, both architectures achieve similar performance and internal geometry, suggesting that explicit modular structure has little impact when representations are weakly constrained. In a lower-dimensional "rich" regime, modularity becomes decisive: the modular network develops graded task-specific subspaces that overlap for similar tasks, partially align for moderately dissimilar tasks, and separate for dissimilar tasks, yielding a more compositional and interpretable organization than the single network. These findings identify the representational regime induced by initialization scale, which co-varies with representational dimensionality, as a key factor governing when compositional, modular structure is functionally beneficial in continual learning, and support viewing safety and robustness as problems of adaptive allocation of representational subspaces rather than fixed separation versus sharing.