# Towards an Improved Metric for Evaluating Disentangled Representations

arXiv: https://arxiv.org/abs/2410.03056 · HTML: https://mlsanity.com/paper/2410.03056/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 6 of 6)

- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Learning Deep Disentangled Embeddings with the F-Statistic Loss](https://mlsanity.com/paper/1802.05312/) (2018)
- [Measuring Disentanglement: A Review of Metrics](https://mlsanity.com/paper/2012.09276/) (2020)
- [Progressive Learning and Disentanglement of Hierarchical Representations](https://mlsanity.com/paper/2002.10549/) (2020)
- [A Preliminary Study of Disentanglement With Insights on the Inadequacy of Metrics](https://mlsanity.com/paper/1911.11791/) (2019)

## Abstract (arXiv)

Disentangled representation learning plays a pivotal role in making representations controllable, interpretable and transferable. Despite its significance in the domain, the quest for reliable and consistent quantitative disentanglement metric remains a major challenge. This stems from the utilisation of diverse metrics measuring different properties and the potential bias introduced by their design. Our work undertakes a comprehensive examination of existing popular disentanglement evaluation metrics, comparing them in terms of measuring aspects of disentanglement (viz. Modularity, Compactness, and Explicitness), detecting the factor-code relationship, and describing the degree of disentanglement. We propose a new framework for quantifying disentanglement, introducing a metric entitled EDI, that leverages the intuitive concept of exclusivity and improved factor-code relationship to minimize ad-hoc decisions. An in-depth analysis reveals that EDI measures essential properties while offering more stability than existing metrics, advocating for its adoption as a standardised approach.