# Measuring Orthogonality in Representations of Generative Models

arXiv: https://arxiv.org/abs/2407.03728 · HTML: https://mlsanity.com/paper/2407.03728/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness](https://mlsanity.com/paper/1811.00007/) (2018)
- [Learning Deep Disentangled Embeddings with the F-Statistic Loss](https://mlsanity.com/paper/1802.05312/) (2018)
- [On Disentangled Representations Learned From Correlated Data](https://mlsanity.com/paper/2006.07886/) (2020)
- [On the Transfer of Disentangled Representations in Realistic Settings](https://mlsanity.com/paper/2010.14407/) (2020)
- [Symmetry-Based Disentangled Representation Learning requires Interaction with Environments](https://mlsanity.com/paper/1904.00243/) (2019)
- [DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability](https://mlsanity.com/paper/2210.00364/) (2022)
- [On Causally Disentangled Representations](https://mlsanity.com/paper/2112.05746/) (2021)
- [Linear Disentangled Representations and Unsupervised Action Estimation](https://mlsanity.com/paper/2008.07922/) (2020)
- [A Deep Network for Joint Registration and Reconstruction of Images with Pathologies](https://mlsanity.com/paper/2008.07628/) (2020)
- [Quantifying and Learning Linear Symmetry-Based Disentanglement](https://mlsanity.com/paper/2011.06070/) (2020)
- [Deep Learning based Multi-modal Computing with Feature Disentanglement for MRI Image Synthesis](https://mlsanity.com/paper/2105.02835/) (2021)
- [Leveraging Relational Information for Learning Weakly Disentangled Representations](https://mlsanity.com/paper/2205.10056/) (2022)
- [Lung Swapping Autoencoder: Learning a Disentangled Structure-texture Representation of Chest Radiographs](https://mlsanity.com/paper/2201.07344/) (2022)
- [Prior image-based medical image reconstruction using a style-based generative adversarial network](https://mlsanity.com/paper/2202.08936/) (2022)
- [Disentanglement enables cross-domain Hippocampus Segmentation](https://mlsanity.com/paper/2201.05650/) (2022)

## Built on by (most cited first, 1 of 1)

- [Do speech foundation models really learn words?](https://mlsanity.com/paper/2609.10434/) (2026)

## Abstract (arXiv)

In unsupervised representation learning, models aim to distill essential features from high-dimensional data into lower-dimensional learned representations, guided by inductive biases. Understanding the characteristics that make a good representation remains a topic of ongoing research. Disentanglement of independent generative processes has long been credited with producing high-quality representations. However, focusing solely on representations that adhere to the stringent requirements of most disentanglement metrics, may result in overlooking many high-quality representations, well suited for various downstream tasks. These metrics often demand that generative factors be encoded in distinct, single dimensions aligned with the canonical basis of the representation space. Motivated by these observations, we propose two novel metrics: Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR). These metrics evaluate the mutual orthogonality and rank of generative factor subspaces. Throughout extensive experiments on common downstream tasks, over several benchmark datasets and models, IWO and IWR consistently show stronger correlations with downstream task performance than traditional disentanglement metrics. Our findings suggest that representation quality is closer related to the orthogonality of independent generative processes rather than their disentanglement, offering a new direction for evaluating and improving unsupervised learning models.