# Disentanglement via Latent Quantization

arXiv: https://arxiv.org/abs/2305.18378 · HTML: https://mlsanity.com/paper/2305.18378/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 80 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 30 of 55)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA](https://mlsanity.com/paper/2107.10098/) (2021)
- [Equinox: neural networks in JAX via callable PyTrees and filtered transformations](https://mlsanity.com/paper/2111.00254/) (2021)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Learning Deep Disentangled Embeddings with the F-Statistic Loss](https://mlsanity.com/paper/1802.05312/) (2018)
- [On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset](https://mlsanity.com/paper/1906.03292/) (2019)
- [Independent mechanism analysis, a new concept?](https://mlsanity.com/paper/2106.05200/) (2021)
- [Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)](https://mlsanity.com/paper/2001.04872/) (2020)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [On the Identifiability of Nonlinear ICA: Sparsity and Beyond](https://mlsanity.com/paper/2206.07751/) (2022)

## Built on by (most cited first, 30 of 40)

- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning](https://mlsanity.com/paper/2505.07538/) (2025)
- [Selective Visual Representations Improve Convergence and Generalization for Embodied AI](https://mlsanity.com/paper/2311.04193/) (2023)
- [Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning](https://mlsanity.com/paper/2404.10282/) (2024)
- [Language-Informed Visual Concept Learning](https://mlsanity.com/paper/2312.03587/) (2023)
- [AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors](https://mlsanity.com/paper/2403.09976/) (2024)
- [Representation Collapsing Problems in Vector Quantization](https://mlsanity.com/paper/2411.16550/) (2024)
- [Discrete Codebook World Models for Continuous Control](https://mlsanity.com/paper/2503.00653/) (2025)
- [From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?](https://mlsanity.com/paper/2512.15134/) (2025)
- [Statistical and structural identifiability in representation learning](https://mlsanity.com/paper/2603.11970/) (2026)
- [Disentangling Representations through Multi-task Learning](https://mlsanity.com/paper/2407.11249/) (2024)
- [Range, not Independence, Drives Modularity in Biologically Inspired Representations](https://mlsanity.com/paper/2410.06232/) (2024)
- [Zero-Shot Generalization of Vision-Based RL Without Data Augmentation](https://mlsanity.com/paper/2410.07441/) (2024)
- [Disentangling Disentangled Representations: Towards Improved Latent Units via Diffusion Models](https://mlsanity.com/paper/2410.23820/) (2024)
- [Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised Code](https://mlsanity.com/paper/2501.18328/) (2025)
- [On the Identifiability of Quantized Factors](https://mlsanity.com/paper/2306.16334/) (2023)
- [Disentangled Representation Learning with the Gromov-Monge Gap](https://mlsanity.com/paper/2407.07829/) (2024)
- [A Revisit of Total Correlation in Disentangled Variational Auto-Encoder with Partial Disentanglement](https://mlsanity.com/paper/2502.02279/) (2025)
- [Zero-Shot Visual Generalization in Robot Manipulation](https://mlsanity.com/paper/2505.11719/) (2025)
- [From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning](https://mlsanity.com/paper/2510.02484/) (2025)
- [Comparing the information content of probabilistic representation spaces](https://mlsanity.com/paper/2405.21042/) (2024)
- [Towards Utilising a Range of Neural Activations for Comprehending Representational Associations](https://mlsanity.com/paper/2411.10019/) (2024)
- [Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models](https://mlsanity.com/paper/2501.01761/) (2025)
- [Disentanglement by means of action-induced representations](https://mlsanity.com/paper/2602.06741/) (2026)
- [Disentanglement with Factor Quantized Variational Autoencoders](https://mlsanity.com/paper/2409.14851/) (2024)
- [Transferring disentangled representations: bridging the gap between synthetic and real images](https://mlsanity.com/paper/2409.18017/) (2024)
- [BrainStratify: Coarse-to-Fine Disentanglement of Intracranial Neural Dynamics](https://mlsanity.com/paper/2505.20480/) (2025)
- [Refining the Information Bottleneck via Adversarial Information Separation](https://mlsanity.com/paper/2602.06549/) (2026)
- [VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbation](https://mlsanity.com/paper/2602.17133/) (2026)

## Abstract (arXiv)

In disentangled representation learning, a model is asked to tease apart a dataset's underlying sources of variation and represent them independently of one another. Since the model is provided with no ground truth information about these sources, inductive biases take a paramount role in enabling disentanglement. In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. Concretely, we do this by (i) quantizing the latent space into discrete code vectors with a separate learnable scalar codebook per dimension and (ii) applying strong model regularization via an unusually high weight decay. Intuitively, the latent space design forces the encoder to combinatorially construct codes from a small number of distinct scalar values, which in turn enables the decoder to assign a consistent meaning to each value. Regularization then serves to drive the model towards this parsimonious strategy. We demonstrate the broad applicability of this approach by adding it to both basic data-reconstructing (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models. For reliable evaluation, we also propose InfoMEC, a new set of metrics for disentanglement that is cohesively grounded in information theory and fixes well-established shortcomings in previous metrics. Together with regularization, latent quantization dramatically improves the modularity and explicitness of learned representations on a representative suite of benchmark datasets. In particular, our quantized-latent autoencoder (QLAE) consistently outperforms strong methods from prior work in these key disentanglement properties without compromising data reconstruction.