# Wasserstein Dependency Measure for Representation Learning

arXiv: https://arxiv.org/abs/1903.11780 · HTML: https://mlsanity.com/paper/1903.11780/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2019.

## Built on by (most cited first, 30 of 69)

- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [Decoupled Contrastive Learning](https://mlsanity.com/paper/2110.06848/) (2021)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [Robust Contrastive Learning against Noisy Views](https://mlsanity.com/paper/2201.04309/) (2022)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Geometric Entropic Exploration](https://mlsanity.com/paper/2101.02055/) (2021)
- [Self-supervised Representation Learning with Relative Predictive Coding](https://mlsanity.com/paper/2103.11275/) (2021)
- [A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning](https://mlsanity.com/paper/2104.13712/) (2021)
- [Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning](https://mlsanity.com/paper/2106.02866/) (2021)
- [Disentangled Recurrent Wasserstein Autoencoder](https://mlsanity.com/paper/2101.07496/) (2021)
- [Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE](https://mlsanity.com/paper/2107.01152/) (2021)
- [Wasserstein Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2110.07940/) (2021)
- [Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning](https://mlsanity.com/paper/2412.08021/) (2024)
- [Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation](https://mlsanity.com/paper/2412.08139/) (2024)
- [Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression](https://mlsanity.com/paper/2305.00869/) (2023)
- [Language Guided Skill Discovery](https://mlsanity.com/paper/2406.06615/) (2024)
- [Improving Multimodal fusion via Mutual Dependency Maximisation](https://mlsanity.com/paper/2109.00922/) (2021)
- [Dynamic Multimodal Information Bottleneck for Multimodality Classification](https://mlsanity.com/paper/2311.01066/) (2023)
- [f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning](https://mlsanity.com/paper/2402.10150/) (2024)
- [TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2206.10698/) (2022)
- [Invariant Learning via Diffusion Dreamed Distribution Shifts](https://mlsanity.com/paper/2211.10370/) (2022)
- [Entropy Regularized Optimal Transport Independence Criterion](https://mlsanity.com/paper/2112.15265/) (2021)
- [Learning Disentangled Behaviour Patterns for Wearable-based Human Activity Recognition](https://mlsanity.com/paper/2202.07260/) (2022)
- [InfoOT: Information Maximizing Optimal Transport](https://mlsanity.com/paper/2210.03164/) (2022)
- [Disentanglement and Generalization Under Correlation Shifts](https://mlsanity.com/paper/2112.14754/) (2021)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [Do's and Don'ts: Learning Desirable Skills with Instruction Videos](https://mlsanity.com/paper/2406.00324/) (2024)
- [Integrating Auxiliary Information in Self-supervised Learning](https://mlsanity.com/paper/2106.02869/) (2021)

## Abstract (arXiv)

Mutual information maximization has emerged as a powerful learning objective for unsupervised representation learning obtaining state-of-the-art performance in applications such as object recognition, speech recognition, and reinforcement learning. However, such approaches are fundamentally limited since a tight lower bound of mutual information requires sample size exponential in the mutual information. This limits the applicability of these approaches for prediction tasks with high mutual information, such as in video understanding or reinforcement learning. In these settings, such techniques are prone to overfit, both in theory and in practice, and capture only a few of the relevant factors of variation. This leads to incomplete representations that are not optimal for downstream tasks. In this work, we empirically demonstrate that mutual information-based representation learning approaches do fail to learn complete representations on a number of designed and real-world tasks. To mitigate these problems we introduce the Wasserstein dependency measure, which learns more complete representations by using the Wasserstein distance instead of the KL divergence in the mutual information estimator. We show that a practical approximation to this theoretically motivated solution, constructed using Lipschitz constraint techniques from the GAN literature, achieves substantially improved results on tasks where incomplete representations are a major challenge.