# Indeterminacy in Generative Models: Characterization and Strong Identifiability

arXiv: https://arxiv.org/abs/2206.00801 · HTML: https://mlsanity.com/paper/2206.00801/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 2 Jun 2022.

## Builds on (most cited first, 25 of 25)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA](https://mlsanity.com/paper/2107.10098/) (2021)
- [Weakly supervised causal representation learning](https://mlsanity.com/paper/2203.16437/) (2022)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [Independent mechanism analysis, a new concept?](https://mlsanity.com/paper/2106.05200/) (2021)
- [Unconstrained Monotonic Neural Networks](https://mlsanity.com/paper/1908.05164/) (2019)
- [The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA](https://mlsanity.com/paper/1905.06642/) (2019)
- [Desiderata for Representation Learning: A Causal Perspective](https://mlsanity.com/paper/2109.03795/) (2021)
- [Weakly Supervised Disentangled Generative Causal Representation Learning](https://mlsanity.com/paper/2010.02637/) (2020)
- [Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA](https://mlsanity.com/paper/2106.09620/) (2021)
- [Sum-of-Squares Polynomial Flow](https://mlsanity.com/paper/1905.02325/) (2019)
- [Causal Discovery with General Non-Linear Relationships Using Non-Linear ICA](https://mlsanity.com/paper/1904.09096/) (2019)
- [Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE](https://mlsanity.com/paper/2011.04798/) (2020)
- [Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning](https://mlsanity.com/paper/2110.15796/) (2021)
- [Triangular Flows for Generative Modeling: Statistical Consistency, Smoothness Classes, and Fast Rates](https://mlsanity.com/paper/2112.15595/) (2021)
- [Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1909.05063/) (2019)

## Built on by (most cited first, 30 of 33)

- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation](https://mlsanity.com/paper/2307.02598/) (2023)
- [Causal Component Analysis](https://mlsanity.com/paper/2305.17225/) (2023)
- [Causal normalizing flows: from theory to practice](https://mlsanity.com/paper/2306.05415/) (2023)
- [Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning](https://mlsanity.com/paper/2303.16535/) (2023)
- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling](https://mlsanity.com/paper/2310.11011/) (2023)
- [Object-centric architectures enable efficient causal representation learning](https://mlsanity.com/paper/2310.19054/) (2023)
- [All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling](https://mlsanity.com/paper/2410.23501/) (2024)
- [Towards Interpretable Deep Generative Models via Causal Representation Learning](https://mlsanity.com/paper/2504.11609/) (2025)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [DeCaFlow: A deconfounding causal generative model](https://mlsanity.com/paper/2503.15114/) (2025)
- [Independent Mechanism Analysis and the Manifold Hypothesis](https://mlsanity.com/paper/2312.13438/) (2023)
- [Robustness of Nonlinear Representation Learning](https://mlsanity.com/paper/2503.15355/) (2025)
- [Propensity Score Alignment of Unpaired Multimodal Data](https://mlsanity.com/paper/2404.01595/) (2024)
- [On the Identifiability of Sparse ICA without Assuming Non-Gaussianity](https://mlsanity.com/paper/2408.10353/) (2024)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Consistent End-to-End Estimation for Counterfactual Fairness](https://mlsanity.com/paper/2310.17687/) (2023)
- [When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective](https://mlsanity.com/paper/2506.03784/) (2025)
- [Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution](https://mlsanity.com/paper/2307.16422/) (2023)
- [Estimating treatment effects from single-arm trials via latent-variable modeling](https://mlsanity.com/paper/2311.03002/) (2023)
- [Causal Order Discovery based on Monotonic SCMs](https://mlsanity.com/paper/2410.19870/) (2024)
- [Nonlinear multi-study sparse factor analysis](https://mlsanity.com/paper/2601.18128/) (2026)
- [A Unifying Framework for Unsupervised Concept Extraction](https://mlsanity.com/paper/2604.24936/) (2026)
- [Identifying latent state transition in non-linear dynamical systems](https://mlsanity.com/paper/2406.03337/) (2024)
- [Unsupervised Object Detection with Theoretical Guarantees](https://mlsanity.com/paper/2406.07284/) (2024)
- [Celcomen: spatial causal disentanglement for single-cell and tissue perturbation modeling](https://mlsanity.com/paper/2409.05804/) (2024)
- [Causally Consistent Normalizing Flow](https://mlsanity.com/paper/2412.12401/) (2024)
- [On Measuring Intrinsic Causal Attributions in Deep Neural Networks](https://mlsanity.com/paper/2505.09660/) (2025)
- [Addressing Correlated Latent Exogenous Variables in Debiased Recommender Systems](https://mlsanity.com/paper/2506.07517/) (2025)

## Abstract (arXiv)

Most modern probabilistic generative models, such as the variational autoencoder (VAE), have certain indeterminacies that are unresolvable even with an infinite amount of data. Different tasks tolerate different indeterminacies, however recent applications have indicated the need for strongly identifiable models, in which an observation corresponds to a unique latent code. Progress has been made towards reducing model indeterminacies while maintaining flexibility, and recent work excludes many--but not all--indeterminacies. In this work, we motivate model-identifiability in terms of task-identifiability, then construct a theoretical framework for analyzing the indeterminacies of latent variable models, which enables their precise characterization in terms of the generator function and prior distribution spaces. We reveal that strong identifiability is possible even with highly flexible nonlinear generators, and give two such examples. One is a straightforward modification of iVAE (arXiv:1907.04809 [stat.ML]); the other uses triangular monotonic maps, leading to novel connections between optimal transport and identifiability.