# A theory of independent mechanisms for extrapolation in generative models

arXiv: https://arxiv.org/abs/2004.00184 · HTML: https://mlsanity.com/paper/2004.00184/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 17 of 17)

- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Visual Representation Learning Does Not Generalize Strongly Within the Same Domain](https://mlsanity.com/paper/2107.08221/) (2021)
- [Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation](https://mlsanity.com/paper/2307.02598/) (2023)
- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Measuring axiomatic soundness of counterfactual image models](https://mlsanity.com/paper/2303.01274/) (2023)
- [Dynamic Inference with Neural Interpreters](https://mlsanity.com/paper/2110.06399/) (2021)
- [Homomorphism Autoencoder -- Learning Group Structured Representations from Observed Transitions](https://mlsanity.com/paper/2207.12067/) (2022)
- [Emerging Synergies in Causality and Deep Generative Models: A Survey](https://mlsanity.com/paper/2301.12351/) (2023)
- [Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imagination](https://mlsanity.com/paper/2311.06673/) (2023)
- [Demystifying Inductive Biases for β-VAE Based Architectures](https://mlsanity.com/paper/2102.06822/) (2021)
- [Going beyond Compositions, DDPMs Can Produce Zero-Shot Interpolations](https://mlsanity.com/paper/2405.19201/) (2024)
- [Cause-effect inference through spectral independence in linear dynamical systems: theoretical foundations](https://mlsanity.com/paper/2110.15595/) (2021)
- [Learning soft interventions in complex equilibrium systems](https://mlsanity.com/paper/2112.05729/) (2021)
- [A Causal Intervention Scheme for Semantic Segmentation of Quasi-periodic Cardiovascular Signals](https://mlsanity.com/paper/2209.09018/) (2022)
- [Graph Community Augmentation with GMM-based Modeling in Latent Space](https://mlsanity.com/paper/2412.01163/) (2024)
- [An Information Theoretic Framework for Graph Novelty Generation via Latent Mixture Modeling](https://mlsanity.com/paper/2606.19770/) (2026)

## Abstract (arXiv)

Generative models can be trained to emulate complex empirical data, but are they useful to make predictions in the context of previously unobserved environments? An intuitive idea to promote such extrapolation capabilities is to have the architecture of such model reflect a causal graph of the true data generating process, such that one can intervene on each node independently of the others. However, the nodes of this graph are usually unobserved, leading to overparameterization and lack of identifiability of the causal structure. We develop a theoretical framework to address this challenging situation by defining a weaker form of identifiability, based on the principle of independence of mechanisms. We demonstrate on toy examples that classical stochastic gradient descent can hinder the model's extrapolation capabilities, suggesting independence of mechanisms should be enforced explicitly during training. Experiments on deep generative models trained on real world data support these insights and illustrate how the extrapolation capabilities of such models can be leveraged.