mlsanityData to 7 Oct 2026

Disentanglement with Biological Constraints: A Theory of Functional Cell Types

arXiv 2210.01768 ↗First posted 30 Sep 2022q-bio.NC cs.LG cs.NEcited by 30

Not on the map. Why?

Abstract (arXiv)

Neurons in the brain are often finely tuned for specific task variables. Moreover, such disentangled representations are highly sought after in machine learning. Here we mathematically prove that simple biological constraints on neurons, namely nonnegativity and energy efficiency in both activity and weights, promote such sought after disentangled representations by enforcing neurons to become selective for single factors of task variation. We demonstrate these constraints lead to disentanglement in a variety of tasks and architectures, including variational autoencoders. We also use this theory to explain why the brain partitions its cells into distinct cell types such as grid and object-vector cells, and also explain when the brain instead entangles representations in response to entangled task factors. Overall, this work provides a mathematical understanding of why single neurons in the brain often represent single human-interpretable factors, and steps towards an understanding task structure shapes the structure of brain representation.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 25 of its 96 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

25 of 96 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  5. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  6. 2018Disentangling by Factorisingcited by 614
  7. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  8. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 240
  9. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  10. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  11. 2018Learning Deep Disentangled Embeddings with the F-Statistic Losscited by 101
  12. 2018Variational Autoencoders Pursue PCA Directions (by Accident)cited by 84
13 more
  1. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  2. 2021Visual Representation Learning Does Not Generalize Strongly Within the Same Domaincited by 60
  3. 2018Emergence of grid-like representations by training recurrent neural networks to perform spatial localizationcited by 52
  4. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 45
  5. 2017Deep Learning Models of the Retinal Response to Natural Scenescited by 36
  6. 2020Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neuronscited by 33
  7. 2015A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Datacited by 26
  8. 2017Blind nonnegative source separation using biological neural networkscited by 11
  9. 2020On Path Integration of Grid Cells: Group Representation and Isotropic Scalingcited by 8
  10. 2020Biologically plausible single-layer networks for nonnegative independent component analysiscited by 8
  11. 2022Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sourcescited by 6
  12. 2021Demystifying Inductive Biases for β-VAE Based Architecturescited by 5
  13. 2021Constellation: Learning relational abstractions over objects for compositional imaginationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 13; 2025: 8; 2026: 4.

30 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2023Disentanglement via Latent Quantizationcited by 40
  3. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  4. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  5. 2024Tripod: Three Complementary Inductive Biases for Disentangled Representation Learningcited by 13
  6. 2025Does Data Scaling Lead to Visual Compositional Generalization?cited by 13
  7. 2023Flow Factorized Representation Learningcited by 11
  8. 2024Poisson Variational Autoencodercited by 11
  9. 2023Probing Biological and Artificial Neural Networks with Task-dependent Neural Manifoldscited by 4
  10. 2024Disentangling Representations through Multi-task Learningcited by 4
  11. 2024Range, not Independence, Drives Modularity in Biologically Inspired Representationscited by 4
  12. 2026Statistical and structural identifiability in representation learningcited by 4
18 more
  1. 2023The Lie-Group Bayesian Learning Rulecited by 3
  2. 2023Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetrycited by 3
  3. 2024Zero-Shot Generalization of Vision-Based RL Without Data Augmentationcited by 3
  4. 2025Zero-Shot Visual Generalization in Robot Manipulationcited by 3
  5. 2025A mathematical theory for understanding when abstract representations emerge in neural networkscited by 3
  6. 2025A Controllable Appearance Representation for Flexible Transfer and Editingcited by 2
  7. 2024Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoningcited by 1
  8. 2024Disentanglement with Factor Quantized Variational Autoencoderscited by 1
  9. 2025Is Generation Required for Data-Efficient Perception?cited by 1
  10. 2026Stimulus symmetries can confound representational similarity analysescited by 1
  11. 2024Disentangled Representations for Causal Cognitioncited by 0
  12. 2024Do Mice Grok? Glimpses of Hidden Progress During Overtraining in Sensory Cortexcited by 0
  13. 2024Disentanglement and Compositionality of Letter Identity and Letter Position in Variational Auto-Encoder Vision Modelscited by 0
  14. 2025Vector Quantization in the Brain: Grid-like Codes in World Modelscited by 0
  15. 2025MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddingscited by 0
  16. 2025Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisationcited by 0
  17. 2026Metabolic cost of information processing in Poisson variational autoencoderscited by 0
  18. 2026The Computational Value of Sensory-Aligned Receptive Fields Depends on Neuronal Expressivitycited by 0