mlsanityData to 14 Oct 2026

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA

arXiv 2107.10098 ↗First posted 21 Jul 2021stat.ML cs.LGcited by 160

Not on the map. Why?

Abstract (arXiv)

This work introduces a novel principle we call disentanglement via mechanism sparsity regularization, which can be applied when the latent factors of interest depend sparsely on past latent factors and/or observed auxiliary variables. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that relates them. We develop a rigorous identifiability theory, building on recent nonlinear independent component analysis (ICA) results, that formalizes this principle and shows how the latent variables can be recovered up to permutation if one regularizes the latent mechanisms to be sparse and if some graph connectivity criterion is satisfied by the data generating process. As a special case of our framework, we show how one can leverage unknown-target interventions on the latent factors to disentangle them, thereby drawing further connections between ICA and causality. We propose a VAE-based method in which the latent mechanisms are learned and regularized via binary masks, and validate our theory by showing it learns disentangled representations in simulations.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

32 of 52 references.

References · 32

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  6. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  7. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  9. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 282
  10. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  11. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 273
  12. 2020Weakly-Supervised Disentanglement Without Compromisescited by 239
20 more
  1. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  2. 2019Recurrent Independent Mechanismscited by 219
  3. 2020Differentiable Causal Discovery from Interventional Datacited by 218
  4. 2021Towards Causal Representation Learningcited by 209
  5. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  6. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  7. 2017CausalGAN: Learning Causal Implicit Generative Models with Adversarial Trainingcited by 152
  8. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 122
  9. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  10. 2017The Consciousness Priorcited by 107
  11. 2020On Linear Identifiability of Learned Representationscited by 105
  12. 2020Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)cited by 92
  13. 2019Masked Gradient-Based Causal Structure Learningcited by 89
  14. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  15. 2019Permutation-Based Causal Structure Learning with Unknown Intervention Targetscited by 69
  16. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  17. 2019Unsupervised Model Selection for Variational Disentangled Representation Learningcited by 46
  18. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  19. 2019Causal Induction from Visual Observations for Goal Directed Taskscited by 31
  20. 2017Independently Controllable Factorscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 47; 2024: 34; 2025: 34; 2026: 28.

160 papers.

Sort:

By area:

  1. 2022Interventional Causal Representation Learningcited by 142
  2. 2022Weakly supervised causal representation learningcited by 140
  3. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 133
  4. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 106
  5. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  6. 2022On the Identifiability of Nonlinear ICA: Sparsity and Beyondcited by 85
  7. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  8. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 74
  9. 2022Learning to Induce Causal Structurecited by 68
  10. 2024Causal Representation Learning from Multiple Distributions: A General Settingcited by 68
  11. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 65
  12. 2023Identifiability Results for Multimodal Contrastive Learningcited by 61
38 more
  1. 2023Identifiability of latent-variable and structural-equation models: from linear to nonlinearcited by 58
  2. 2023Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolationcited by 58
  3. 2023Causal Component Analysiscited by 56
  4. 2022Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesiscited by 55
  5. 2021Identifiable Deep Generative Models via Sparse Decodingcited by 54
  6. 2022Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systemscited by 51
  7. 2023Score-based Causal Representation Learning with Interventionscited by 49
  8. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
  9. 2023Provably Learning Object-Centric Representationscited by 45
  10. 2022Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learningcited by 44
  11. 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
  12. 2023Disentanglement via Latent Quantizationcited by 40
  13. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  14. 2023BISCUIT: Causal Representation Learning from Binary Interactionscited by 38
  15. 2023Modelling Cellular Perturbations with the Sparse Additive Mechanism Shift Variational Autoencodercited by 38
  16. 2024A Sparsity Principle for Partially Observable Causal Representation Learningcited by 35
  17. 2024PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysiscited by 35
  18. 2022Identifying Weight-Variant Latent Causal Modelscited by 34
  19. 2022Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modelingcited by 34
  20. 2023Learning nonparametric latent causal graphs with unknown interventionscited by 34
  21. 2023General Identifiability and Achievability for Causal Representation Learningcited by 34
  22. 2023Generalizing Nonlinear ICA Beyond Structural Sparsitycited by 34
  23. 2022Indeterminacy in Generative Models: Characterization and Strong Identifiabilitycited by 33
  24. 2023Leveraging sparse and shared feature activations for disentangled representation learningcited by 32
  25. 2023Causal Representation Learning Made Identifiable by Grouping of Observational Variablescited by 32
  26. 2024Unifying Causal Representation Learning with the Invariance Principlecited by 31
  27. 2025Thought Communication in Multiagent Collaborationcited by 30
  28. 2026When Does LeJEPA Learn a World Model?cited by 30
  29. 2022Partial Disentanglement via Mechanism Sparsitycited by 29
  30. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  31. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  32. 2023Identification of Nonlinear Latent Hierarchical Modelscited by 27
  33. 2023Learning World Models with Identifiable Factorizationcited by 23
  34. 2023Multi-Domain Causal Representation Learning via Weak Distributional Invariancescited by 23
  35. 2023Identifiable Latent Polynomial Causal Models Through the Lens of Changecited by 23
  36. 2024CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Processcited by 23
  37. 2024Marrying Causal Representation Learning with Dynamical Systems for Sciencecited by 23
  38. 2023Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learningcited by 22

50 most cited of 160 All 160 (JSON)