# Multi-Facet Clustering Variational Autoencoders

arXiv: https://arxiv.org/abs/2106.05241 · HTML: https://mlsanity.com/paper/2106.05241/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 77 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 30 of 33)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [SpectralNet: Spectral Clustering using Deep Neural Networks](https://mlsanity.com/paper/1801.01587/) (2018)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [ClusterGAN : Latent Space Clustering in Generative Adversarial Networks](https://mlsanity.com/paper/1809.03627/) (2018)
- [Deep Spectral Clustering using Dual Autoencoder Network](https://mlsanity.com/paper/1904.13113/) (2019)
- [Clustering with Deep Learning: Taxonomy and New Methods](https://mlsanity.com/paper/1801.07648/) (2018)
- [Progressive Learning and Disentanglement of Hierarchical Representations](https://mlsanity.com/paper/2002.10549/) (2020)
- [Data-dependent Initializations of Convolutional Neural Networks](https://mlsanity.com/paper/1511.06856/) (2015)
- [Nonparametric Variational Auto-encoders for Hierarchical Representation Learning](https://mlsanity.com/paper/1703.07027/) (2017)

## Built on by (most cited first, 28 of 28)

- [Learning Linear Causal Representations from Interventions under General Nonlinear Mixing](https://mlsanity.com/paper/2306.02235/) (2023)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Identifiability of deep generative models without auxiliary information](https://mlsanity.com/paper/2206.10044/) (2022)
- [Generalizing Nonlinear ICA Beyond Structural Sparsity](https://mlsanity.com/paper/2311.00866/) (2023)
- [Causal Representation Learning Made Identifiable by Grouping of Observational Variables](https://mlsanity.com/paper/2310.15709/) (2023)
- [Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models](https://mlsanity.com/paper/2402.09236/) (2024)
- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [Manifold Learning by Mixture Models of VAEs for Inverse Problems](https://mlsanity.com/paper/2303.15244/) (2023)
- [A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEs](https://mlsanity.com/paper/2301.08187/) (2023)
- [Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention](https://mlsanity.com/paper/2406.07141/) (2024)
- [The Gaussian Discriminant Variational Autoencoder (GdVAE): A Self-Explainable Model with Counterfactual Explanations](https://mlsanity.com/paper/2409.12952/) (2024)
- [Long-horizon video prediction using a dynamic latent hierarchy](https://mlsanity.com/paper/2212.14376/) (2022)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Deep Taxonomic Networks for Unsupervised Hierarchical Prototype Discovery](https://mlsanity.com/paper/2509.23602/) (2025)
- [PDGMM-VAE: A Variational Autoencoder with Adaptive Per-Dimension Gaussian Mixture Model Priors for Nonlinear ICA](https://mlsanity.com/paper/2603.23547/) (2026)
- [A Causal Ordering Prior for Unsupervised Representation Learning](https://mlsanity.com/paper/2307.05704/) (2023)
- [Generalized Identifiability Bounds for Mixture Models with Grouped Samples](https://mlsanity.com/paper/2207.11164/) (2022)
- [A Prototype-Oriented Clustering for Domain Shift with Source Privacy](https://mlsanity.com/paper/2302.03807/) (2023)
- [Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior](https://mlsanity.com/paper/2410.10180/) (2024)
- [Causal Dynamic Variational Autoencoder for Counterfactual Regression in Longitudinal Data](https://mlsanity.com/paper/2310.10559/) (2023)
- [Estimating Dataset Dimension via Singular Metrics under the Manifold Hypothesis: Application to Inverse Problems](https://mlsanity.com/paper/2507.07291/) (2025)
- [Heavy-Tailed Class-Conditional Priors for Long-Tailed Generative Modeling](https://mlsanity.com/paper/2509.02154/) (2025)
- [PrismQuant: Rate-Distortion-Optimal Vector Quantization for Gaussian-Mixture Sources](https://mlsanity.com/paper/2605.15507/) (2026)
- [A generative recommender system with GMM prior for cancer drug generation and sensitivity prediction](https://mlsanity.com/paper/2206.03555/) (2022)
- [CAVACHON: a hierarchical variational autoencoder to integrate multi-modal single-cell data](https://mlsanity.com/paper/2405.18655/) (2024)
- [An Adaptor for Triggering Semi-Supervised Learning to Out-of-Box Serve Deep Image Clustering](https://mlsanity.com/paper/2509.20976/) (2025)
- [Structured Coupling for Flow Matching](https://mlsanity.com/paper/2605.07676/) (2026)
- [Beyond Independence: Learning Correlated Views for Variational Incomplete Multi-View Clustering](https://mlsanity.com/paper/2608.15757/) (2026)

## Abstract (arXiv)

Work in deep clustering focuses on finding a single partition of data. However, high-dimensional data, such as images, typically feature multiple interesting characteristics one could cluster over. For example, images of objects against a background could be clustered over the shape of the object and separately by the colour of the background. In this paper, we introduce Multi-Facet Clustering Variational Autoencoders (MFCVAE), a novel class of variational autoencoders with a hierarchy of latent variables, each with a Mixture-of-Gaussians prior, that learns multiple clusterings simultaneously, and is trained fully unsupervised and end-to-end. MFCVAE uses a progressively-trained ladder architecture which leads to highly stable performance. We provide novel theoretical results for optimising the ELBO analytically with respect to the categorical variational posterior distribution, correcting earlier influential theoretical work. On image benchmarks, we demonstrate that our approach separates out and clusters over different aspects of the data in a disentangled manner. We also show other advantages of our model: the compositionality of its latent space and that it provides controlled generation of samples.