# Discovering Hidden Factors of Variation in Deep Networks

arXiv: https://arxiv.org/abs/1412.6583 · HTML: https://mlsanity.com/paper/1412.6583/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2014.

## Built on by (most cited first, 21 of 21)

- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations](https://mlsanity.com/paper/2203.02013/) (2022)
- [An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild](https://mlsanity.com/paper/2106.15610/) (2021)
- [Improving Multi-Task Generalization via Regularizing Spurious Correlation](https://mlsanity.com/paper/2205.09797/) (2022)
- [Symmetry-Based Representations for Artificial and Biological General Intelligence](https://mlsanity.com/paper/2203.09250/) (2022)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobao](https://mlsanity.com/paper/2306.03527/) (2023)
- [Scalable Infomin Learning](https://mlsanity.com/paper/2302.10701/) (2023)
- [Novel View Synthesis via Depth-guided Skip Connections](https://mlsanity.com/paper/2101.01619/) (2021)
- [Disentangled Representations for Short-Term and Long-Term Person Re-Identification](https://mlsanity.com/paper/2409.05277/) (2024)
- [Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval](https://mlsanity.com/paper/2103.12328/) (2021)
- [Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research](https://mlsanity.com/paper/2301.08525/) (2023)
- [Towards Controllable Time Series Generation](https://mlsanity.com/paper/2403.03698/) (2024)
- [Place Recognition under Occlusion and Changing Appearance via Disentangled Representations](https://mlsanity.com/paper/2211.11439/) (2022)
- [SCADI: Self-supervised Causal Disentanglement in Latent Variable Models](https://mlsanity.com/paper/2311.06567/) (2023)
- [How Video Meetings Change Your Expression](https://mlsanity.com/paper/2406.00955/) (2024)
- [Enhancing Interpretability in Generative Modeling: Statistically Disentangled Latent Spaces Guided by Generative Factors in Scientific Datasets](https://mlsanity.com/paper/2507.00298/) (2025)
- [Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts](https://mlsanity.com/paper/2311.10177/) (2023)
- [Disentangling stellar atmospheric parameters in astronomical spectra using Generative Adversarial Neural Networks](https://mlsanity.com/paper/2501.11762/) (2025)
- [From Points to Spheres: A Geometric Reinterpretation of Variational Autoencoders](https://mlsanity.com/paper/2507.17255/) (2025)
- [What We Don't C: Manifold Disentanglement for Structured Discovery](https://mlsanity.com/paper/2511.09433/) (2025)

## Abstract (arXiv)

Deep learning has enjoyed a great deal of success because of its ability to learn useful features for tasks such as classification. But there has been less exploration in learning the factors of variation apart from the classification signal. By augmenting autoencoders with simple regularization terms during training, we demonstrate that standard deep architectures can discover and explicitly represent factors of variation beyond those relevant for categorization. We introduce a cross-covariance penalty (XCov) as a method to disentangle factors like handwriting style for digits and subject identity in faces. We demonstrate this on the MNIST handwritten digit database, the Toronto Faces Database (TFD) and the Multi-PIE dataset by generating manipulated instances of the data. Furthermore, we demonstrate these deep networks can extrapolate `hidden' variation in the supervised signal.