# Range, not Independence, Drives Modularity in Biologically Inspired Representations

arXiv: https://arxiv.org/abs/2410.06232 · HTML: https://mlsanity.com/paper/2410.06232/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 66 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [On the Identifiability of Nonlinear ICA: Sparsity and Beyond](https://mlsanity.com/paper/2206.07751/) (2022)
- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Disentanglement of Correlated Factors via Hausdorff Factorized Support](https://mlsanity.com/paper/2210.07347/) (2022)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [Conditional Mutual Information for Disentangled Representations in Reinforcement Learning](https://mlsanity.com/paper/2305.14133/) (2023)
- [Discovering modular solutions that generalize compositionally](https://mlsanity.com/paper/2312.15001/) (2023)
- [Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning](https://mlsanity.com/paper/2402.18361/) (2024)
- [Polytopic Matrix Factorization: Determinant Maximization Based Criterion and Identifiability](https://mlsanity.com/paper/2202.09638/) (2022)
- [On The Specialization of Neural Modules](https://mlsanity.com/paper/2409.14981/) (2024)
- [Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning](https://mlsanity.com/paper/2404.10282/) (2024)
- [The Sense of Place: Grid Cells in the Brain and the Transcendental Number e](https://mlsanity.com/paper/1304.0031/) (2013)
- [Blind nonnegative source separation using biological neural networks](https://mlsanity.com/paper/1706.00382/) (2017)
- [Actionable Neural Representations: Grid Cells from Minimal Constraints](https://mlsanity.com/paper/2209.15563/) (2022)
- [Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources](https://mlsanity.com/paper/2209.12894/) (2022)

## Built on by (most cited first, 4 of 4)

- [A mathematical theory for understanding when abstract representations emerge in neural networks](https://mlsanity.com/paper/2510.09816/) (2025)
- [Stimulus symmetries can confound representational similarity analyses](https://mlsanity.com/paper/2605.21324/) (2026)
- [Integrative neurocybernetic modeling in the era of large-scale neuroscience](https://mlsanity.com/paper/2604.23903/) (2026)
- [How Optimality Structures Sparse Dictionaries: Theory for Interpreting SAE Representations](https://mlsanity.com/paper/2606.02385/) (2026)

## Abstract (arXiv)

Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired networks -- those that are nonnegative and energy efficient -- modularise their representation of source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather we show that sources modularise if their support is ``sufficiently spread''. From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data, showing that range independence can be used to understand the mixing or modularising of spatial and reward information in entorhinal recordings in seemingly conflicting experiments. Further, we use these results to suggest alternate origins of mixed-selectivity, beyond the predominant theory of flexible nonlinear classification. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.