# Learning Invariant Representations using Inverse Contrastive Loss

arXiv: https://arxiv.org/abs/2102.08343 · HTML: https://mlsanity.com/paper/2102.08343/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 16 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Empirical Risk Minimization under Fairness Constraints](https://mlsanity.com/paper/1802.08626/) (2018)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Invariant Representations without Adversarial Training](https://mlsanity.com/paper/1805.09458/) (2018)
- [Scanner Invariant Representations for Diffusion MRI Harmonization](https://mlsanity.com/paper/1904.05375/) (2019)
- [FairALM: Augmented Lagrangian Method for Training Fair Models with Little Regret](https://mlsanity.com/paper/2004.01355/) (2020)
- [Unsupervised Adversarial Invariance](https://mlsanity.com/paper/1809.10083/) (2018)
- [Learning unbiased features](https://mlsanity.com/paper/1412.5244/) (2014)
- [Unified Adversarial Invariance](https://mlsanity.com/paper/1905.03629/) (2019)

## Built on by (most cited first, 6 of 6)

- [On the Versatile Uses of Partial Distance Correlation in Deep Learning](https://mlsanity.com/paper/2207.09684/) (2022)
- [Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets](https://mlsanity.com/paper/2203.15234/) (2022)
- [Intrinsic Anomaly Detection for Multi-Variate Time Series](https://mlsanity.com/paper/2206.14342/) (2022)
- [Is Exchangeability better than I.I.D to handle Data Distribution Shifts while Pooling Data for Data-scarce Medical image segmentation?](https://mlsanity.com/paper/2507.19575/) (2025)
- [Nearest-Neighbor Density Estimation for Dependency Suppression](https://mlsanity.com/paper/2603.04224/) (2026)
- [PiRL: Participant-Invariant Representation Learning for Healthcare](https://mlsanity.com/paper/2211.12422/) (2022)

## Abstract (arXiv)

Learning invariant representations is a critical first step in a number of machine learning tasks. A common approach corresponds to the so-called information bottleneck principle in which an application dependent function of mutual information is carefully chosen and optimized. Unfortunately, in practice, these functions are not suitable for optimization purposes since these losses are agnostic of the metric structure of the parameters of the model. We introduce a class of losses for learning representations that are invariant to some extraneous variable of interest by inverting the class of contrastive losses, i.e., inverse contrastive loss (ICL). We show that if the extraneous variable is binary, then optimizing ICL is equivalent to optimizing a regularized MMD divergence. More generally, we also show that if we are provided a metric on the sample space, our formulation of ICL can be decomposed into a sum of convex functions of the given distance metric. Our experimental results indicate that models obtained by optimizing ICL achieve significantly better invariance to the extraneous variable for a fixed desired level of accuracy. In a variety of experimental settings, we show applicability of ICL for learning invariant representations for both continuous and discrete extraneous variables.