# Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA

arXiv: https://arxiv.org/abs/2107.10098 · HTML: https://mlsanity.com/paper/2107.10098/

## Key facts

- Cited by 160 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 30 of 32)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Differentiable Causal Discovery from Interventional Data](https://mlsanity.com/paper/2007.01754/) (2020)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [Learning Neural Causal Models from Unknown Interventions](https://mlsanity.com/paper/1910.01075/) (2019)
- [The Consciousness Prior](https://mlsanity.com/paper/1709.08568/) (2017)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)](https://mlsanity.com/paper/2001.04872/) (2020)
- [Masked Gradient-Based Causal Structure Learning](https://mlsanity.com/paper/1910.08527/) (2019)
- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Permutation-Based Causal Structure Learning with Unknown Intervention Targets](https://mlsanity.com/paper/1910.09007/) (2019)
- [Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning](https://mlsanity.com/paper/2107.00848/) (2021)
- [Unsupervised Model Selection for Variational Disentangled Representation Learning](https://mlsanity.com/paper/1905.12614/) (2019)
- [Fast and Slow Learning of Recurrent Independent Mechanisms](https://mlsanity.com/paper/2105.08710/) (2021)

## Built on by (most cited first, 30 of 160)

- [Interventional Causal Representation Learning](https://mlsanity.com/paper/2209.11924/) (2022)
- [Weakly supervised causal representation learning](https://mlsanity.com/paper/2203.16437/) (2022)
- [CITRIS: Causal Identifiability from Temporal Intervened Sequences](https://mlsanity.com/paper/2202.03169/) (2022)
- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Learning Linear Causal Representations from Interventions under General Nonlinear Mixing](https://mlsanity.com/paper/2306.02235/) (2023)
- [On the Identifiability of Nonlinear ICA: Sparsity and Beyond](https://mlsanity.com/paper/2206.07751/) (2022)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Multi-View Causal Representation Learning with Partial Observability](https://mlsanity.com/paper/2311.04056/) (2023)
- [Learning to Induce Causal Structure](https://mlsanity.com/paper/2204.04875/) (2022)
- [Causal Representation Learning from Multiple Distributions: A General Setting](https://mlsanity.com/paper/2402.05052/) (2024)
- [Weakly Supervised Representation Learning with Sparse Perturbations](https://mlsanity.com/paper/2206.01101/) (2022)
- [Identifiability Results for Multimodal Contrastive Learning](https://mlsanity.com/paper/2303.09166/) (2023)
- [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023)
- [Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation](https://mlsanity.com/paper/2307.02598/) (2023)
- [Causal Component Analysis](https://mlsanity.com/paper/2305.17225/) (2023)
- [Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesis](https://mlsanity.com/paper/2206.02013/) (2022)
- [Identifiable Deep Generative Models via Sparse Decoding](https://mlsanity.com/paper/2110.10804/) (2021)
- [Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems](https://mlsanity.com/paper/2206.06169/) (2022)
- [Score-based Causal Representation Learning with Interventions](https://mlsanity.com/paper/2301.08230/) (2023)
- [Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning](https://mlsanity.com/paper/2303.16535/) (2023)
- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning](https://mlsanity.com/paper/2211.14666/) (2022)
- [Provable Compositional Generalization for Object-Centric Learning](https://mlsanity.com/paper/2310.05327/) (2023)
- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [Understanding Masked Autoencoders via Hierarchical Latent Variable Models](https://mlsanity.com/paper/2306.04898/) (2023)
- [BISCUIT: Causal Representation Learning from Binary Interactions](https://mlsanity.com/paper/2306.09643/) (2023)
- [Modelling Cellular Perturbations with the Sparse Additive Mechanism Shift Variational Autoencoder](https://mlsanity.com/paper/2311.02794/) (2023)
- [A Sparsity Principle for Partially Observable Causal Representation Learning](https://mlsanity.com/paper/2403.08335/) (2024)
- [PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis](https://mlsanity.com/paper/2408.10609/) (2024)
- [Identifying Weight-Variant Latent Causal Models](https://mlsanity.com/paper/2208.14153/) (2022)

## Abstract (arXiv)

This work introduces a novel principle we call disentanglement via mechanism sparsity regularization, which can be applied when the latent factors of interest depend sparsely on past latent factors and/or observed auxiliary variables. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that relates them. We develop a rigorous identifiability theory, building on recent nonlinear independent component analysis (ICA) results, that formalizes this principle and shows how the latent variables can be recovered up to permutation if one regularizes the latent mechanisms to be sparse and if some graph connectivity criterion is satisfied by the data generating process. As a special case of our framework, we show how one can leverage unknown-target interventions on the latent factors to disentangle them, thereby drawing further connections between ICA and causality. We propose a VAE-based method in which the latent mechanisms are learned and regularized via binary masks, and validate our theory by showing it learns disentangled representations in simulations.