# Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation

arXiv: https://arxiv.org/abs/2108.07668 · HTML: https://mlsanity.com/paper/2108.07668/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)
- [Domain Agnostic Learning with Disentangled Representations](https://mlsanity.com/paper/1904.12347/) (2019)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [Deep Learning of Representations: Looking Forward](https://mlsanity.com/paper/1305.0445/) (2013)
- [Learning Disentangled Joint Continuous and Discrete Representations](https://mlsanity.com/paper/1804.00104/) (2018)
- [Guided Variational Autoencoder for Disentanglement Learning](https://mlsanity.com/paper/2004.01255/) (2020)
- [Unsupervised Domain Adaptation via Disentangled Representations: Application to Cross-Modality Liver Segmentation](https://mlsanity.com/paper/1907.13590/) (2019)
- [Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders](https://mlsanity.com/paper/1804.10469/) (2018)
- [Regularizing CNNs with Locally Constrained Decorrelations](https://mlsanity.com/paper/1611.01967/) (2016)
- [The Geometry of Deep Generative Image Models and its Applications](https://mlsanity.com/paper/2101.06006/) (2021)
- [A Spectral Regularizer for Unsupervised Disentanglement](https://mlsanity.com/paper/1812.01161/) (2018)
- [Improving training of deep neural networks via Singular Value Bounding](https://mlsanity.com/paper/1611.06013/) (2016)
- [Learning Discrete and Continuous Factors of Data via Alternating Disentanglement](https://mlsanity.com/paper/1905.09432/) (2019)

## Built on by (most cited first, 30 of 35)

- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis](https://mlsanity.com/paper/2211.14506/) (2022)
- [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023)
- [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2203.15799/) (2022)
- [Latent Traversals in Generative Models as Potential Flows](https://mlsanity.com/paper/2304.12944/) (2023)
- [Attribute Group Editing for Reliable Few-shot Image Generation](https://mlsanity.com/paper/2203.08422/) (2022)
- [LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis](https://mlsanity.com/paper/2301.04604/) (2023)
- [When are Post-hoc Conceptual Explanations Identifiable?](https://mlsanity.com/paper/2206.13872/) (2022)
- [TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction](https://mlsanity.com/paper/2502.10982/) (2025)
- [Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning](https://mlsanity.com/paper/2404.10282/) (2024)
- [Flow Factorized Representation Learning](https://mlsanity.com/paper/2309.13167/) (2023)
- [CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration](https://mlsanity.com/paper/2306.08226/) (2023)
- [IA-FaceS: A Bidirectional Method for Semantic Face Editing](https://mlsanity.com/paper/2203.13097/) (2022)
- [Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generation](https://mlsanity.com/paper/2208.12550/) (2022)
- [DAVA: Disentangling Adversarial Variational Autoencoder](https://mlsanity.com/paper/2303.01384/) (2023)
- [Stable Attribute Group Editing for Reliable Few-shot Image Generation](https://mlsanity.com/paper/2302.00179/) (2023)
- [Probing the Robustness of Independent Mechanism Analysis for Representation Learning](https://mlsanity.com/paper/2207.06137/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Householder Projector for Unsupervised Latent Semantics Discovery](https://mlsanity.com/paper/2307.08012/) (2023)
- [Principal Manifold Flows](https://mlsanity.com/paper/2202.07037/) (2022)
- [Disentanglement Learning via Topology](https://mlsanity.com/paper/2308.12696/) (2023)
- [Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability](https://mlsanity.com/paper/2510.17040/) (2025)
- [JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation](https://mlsanity.com/paper/2409.12156/) (2024)
- [Cross-GAN Auditing: Unsupervised Identification of Attribute Level Similarities and Differences between Pretrained Generative Models](https://mlsanity.com/paper/2303.10774/) (2023)
- [Unsupervised Representation Learning from Sparse Transformation Analysis](https://mlsanity.com/paper/2410.05564/) (2024)
- [Content-Style Identification via Differential Independence](https://mlsanity.com/paper/2605.17827/) (2026)
- [Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation](https://mlsanity.com/paper/2202.13363/) (2022)
- [A Survey on Leveraging Pre-trained Generative Adversarial Networks for Image Editing and Restoration](https://mlsanity.com/paper/2207.10309/) (2022)
- [Variantional autoencoder with decremental information bottleneck for disentanglement](https://mlsanity.com/paper/2303.12959/) (2023)
- [Batch Normalization in Cytometry Data by kNN-Graph Preservation](https://mlsanity.com/paper/2304.00050/) (2023)

## Abstract (arXiv)

Unsupervised disentanglement learning is a crucial issue for understanding and exploiting deep generative models. Recently, SeFa tries to find latent disentangled directions by performing SVD on the first projection of a pre-trained GAN. However, it is only applied to the first layer and works in a post-processing way. Hessian Penalty minimizes the off-diagonal entries of the output's Hessian matrix to facilitate disentanglement, and can be applied to multi-layers.However, it constrains each entry of output independently, making it not sufficient in disentangling the latent directions (e.g., shape, size, rotation, etc.) of spatially correlated variations. In this paper, we propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. It simply encourages the variation of output caused by perturbations on different latent dimensions to be orthogonal, and the Jacobian with respect to the input is calculated to represent this variation. We show that our OroJaR also encourages the output's Hessian matrix to be diagonal in an indirect manner. In contrast to the Hessian Penalty, our OroJaR constrains the output in a holistic way, making it very effective in disentangling latent dimensions corresponding to spatially correlated variations. Quantitative and qualitative experimental results show that our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods. Our code is available at https://github.com/csyxwei/OroJaR