# XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision

arXiv: https://arxiv.org/abs/2601.21688 · HTML: https://mlsanity.com/paper/2601.21688/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 28 of 28)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [Counterfactual Generative Networks](https://mlsanity.com/paper/2101.06046/) (2021)
- [On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset](https://mlsanity.com/paper/1906.03292/) (2019)
- [Diffusion Visual Counterfactual Explanations](https://mlsanity.com/paper/2210.11841/) (2022)
- [On Disentangled Representations Learned From Correlated Data](https://mlsanity.com/paper/2006.07886/) (2020)
- [Disentangling factors of variation in deep representations using adversarial training](https://mlsanity.com/paper/1611.03383/) (2016)
- [Learning Disentangled Representation by Exploiting Pretrained Generative Models: A Contrastive Learning View](https://mlsanity.com/paper/2102.10543/) (2021)
- [DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models](https://mlsanity.com/paper/2301.13721/) (2023)
- [Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations](https://mlsanity.com/paper/1811.04784/) (2018)
- [Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement](https://mlsanity.com/paper/2402.09712/) (2024)
- [Disentangled Information Bottleneck](https://mlsanity.com/paper/2012.07372/) (2020)
- [Dual Swap Disentangling](https://mlsanity.com/paper/1805.10583/) (2018)
- [An Empirical Study on Disentanglement of Negative-free Contrastive Learning](https://mlsanity.com/paper/2206.04756/) (2022)
- [Disentanglement and Generalization Under Correlation Shifts](https://mlsanity.com/paper/2112.14754/) (2021)
- [SW-VAE: Weakly Supervised Learn Disentangled Representation Via Latent Factor Swapping](https://mlsanity.com/paper/2209.10623/) (2022)

## Abstract (arXiv)

Disentangled representation learning aims to map independent factors of variation to independent representation components. On one hand, purely unsupervised approaches have proven successful on fully disentangled synthetic data, but fail to recover semantic factors from real data without strong inductive biases. On the other hand, supervised approaches are unstable and hard to scale to large attribute sets because they rely on adversarial objectives or auxiliary classifiers. We introduce textscXFactors, a weakly-supervised VAE framework that disentangles and provides explicit control over a chosen set of factors. Building on the Disentangled Information Bottleneck perspective, we decompose the representation into a residual subspace S and factor-specific subspaces T1,ldots,TK and a residual subspace S. Each target factor is encoded in its assigned Ti through contrastive supervision: an InfoNCE loss pulls together latents sharing the same factor value and pushes apart mismatched pairs. In parallel, KL regularization imposes a Gaussian structure on both S and the aggregated factor subspaces, organizing the geometry without additional supervision for non-targeted factors and avoiding adversarial training and classifiers. Across multiple datasets, with constant hyperparameters, textscXFactors achieves state-of-the-art disentanglement scores and yields consistent qualitative factor alignment in the corresponding subspaces, enabling controlled factor swapping via latent replacement. We further demonstrate that our method scales correctly with increasing latent capacity and evaluate it on the real-world dataset CelebA. Our code is available at hrefhttps://github.com/ICML26-anon/XFactorsgithub.com/ICML26-anon/XFactors.