# Toward the Identifiability of Comparative Deep Generative Models

arXiv: https://arxiv.org/abs/2401.15903 · HTML: https://mlsanity.com/paper/2401.15903/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 69 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 30 of 30)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems](https://mlsanity.com/paper/1906.00331/) (2019)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA](https://mlsanity.com/paper/2107.10098/) (2021)
- [Interventional Causal Representation Learning](https://mlsanity.com/paper/2209.11924/) (2022)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Learning Linear Causal Representations from Interventions under General Nonlinear Mixing](https://mlsanity.com/paper/2306.02235/) (2023)
- [Identifiability Guarantees for Causal Disentanglement from Soft Interventions](https://mlsanity.com/paper/2307.06250/) (2023)
- [Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation](https://mlsanity.com/paper/2307.02598/) (2023)
- [Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling](https://mlsanity.com/paper/2211.03553/) (2022)
- [Learning nonparametric latent causal graphs with unknown interventions](https://mlsanity.com/paper/2306.02899/) (2023)
- [Information Constraints on Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1805.08672/) (2018)
- [Unpaired Multi-Domain Causal Representation Learning](https://mlsanity.com/paper/2302.00993/) (2023)
- [Partial Disentanglement via Mechanism Sparsity](https://mlsanity.com/paper/2207.07732/) (2022)
- [Contrastive Variational Autoencoder Enhances Salient Features](https://mlsanity.com/paper/1902.04601/) (2019)
- [Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints](https://mlsanity.com/paper/2208.04425/) (2022)
- [Unsupervised learning with contrastive latent variable models](https://mlsanity.com/paper/1811.06094/) (2018)
- [Moment Matching Deep Contrastive Latent Variable Models](https://mlsanity.com/paper/2202.10560/) (2022)
- [Probabilistic Contrastive Principal Component Analysis](https://mlsanity.com/paper/2012.07977/) (2020)
- [SepVAE: a contrastive VAE to separate pathological patterns from healthy ones](https://mlsanity.com/paper/2307.06206/) (2023)
- [Contrastive latent variable modeling with application to case-control sequencing experiments](https://mlsanity.com/paper/2102.06731/) (2021)
- [Learning Disentangled Representations with Reference-Based Variational Autoencoders](https://mlsanity.com/paper/1901.08534/) (2019)

## Built on by (most cited first, 4 of 4)

- [Learning Identifiable Factorized Causal Representations of Cellular Responses](https://mlsanity.com/paper/2410.22472/) (2024)
- [Contrastive independent component analysis](https://mlsanity.com/paper/2407.02357/) (2024)
- [Supervised Contrastive Block Disentanglement](https://mlsanity.com/paper/2502.07281/) (2025)
- [Beyond ICA: Identifiability by Symmetry Breaking](https://mlsanity.com/paper/2607.23182/) (2026)

## Abstract (arXiv)

Deep Generative Models (DGMs) are versatile tools for learning data representations while adequately incorporating domain knowledge such as the specification of conditional probability distributions. Recently proposed DGMs tackle the important task of comparing data sets from different sources. One such example is the setting of contrastive analysis that focuses on describing patterns that are enriched in a target data set compared to a background data set. The practical deployment of those models often assumes that DGMs naturally infer interpretable and modular latent representations, which is known to be an issue in practice. Consequently, existing methods often rely on ad-hoc regularization schemes, although without any theoretical grounding. Here, we propose a theory of identifiability for comparative DGMs by extending recent advances in the field of non-linear independent component analysis. We show that, while these models lack identifiability across a general class of mixing functions, they surprisingly become identifiable when the mixing function is piece-wise affine (e.g., parameterized by a ReLU neural network). We also investigate the impact of model misspecification, and empirically show that previously proposed regularization techniques for fitting comparative DGMs help with identifiability when the number of latent variables is not known in advance. Finally, we introduce a novel methodology for fitting comparative DGMs that improves the treatment of multiple data sources via multi-objective optimization and that helps adjust the hyperparameter for the regularization in an interpretable manner, using constrained optimization. We empirically validate our theory and new methodology using simulated data as well as a recent data set of genetic perturbations in cells profiled via single-cell RNA sequencing.