# Invariant Representations without Adversarial Training

arXiv: https://arxiv.org/abs/1805.09458 · HTML: https://mlsanity.com/paper/1805.09458/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 30 of 112)

- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Desiderata for Representation Learning: A Causal Perspective](https://mlsanity.com/paper/2109.03795/) (2021)
- [Controllable Guarantees for Fair Outcomes via Contrastive Information Estimation](https://mlsanity.com/paper/2101.04108/) (2021)
- [Fair Normalizing Flows](https://mlsanity.com/paper/2106.05937/) (2021)
- [An Empirical Framework for Domain Generalization in Clinical Settings](https://mlsanity.com/paper/2103.11163/) (2021)
- [Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization](https://mlsanity.com/paper/2108.05449/) (2021)
- [Survey of Social Bias in Vision-Language Models](https://mlsanity.com/paper/2309.14381/) (2023)
- [An Information-theoretic Approach to Distribution Shifts](https://mlsanity.com/paper/2106.03783/) (2021)
- [Efficient fair PCA for fair representation learning](https://mlsanity.com/paper/2302.13319/) (2023)
- [Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE](https://mlsanity.com/paper/2107.01152/) (2021)
- [FARE: Provably Fair Representation Learning with Practical Certificates](https://mlsanity.com/paper/2210.07213/) (2022)
- [Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation](https://mlsanity.com/paper/2103.12857/) (2021)
- [On the Versatile Uses of Partial Distance Correlation in Deep Learning](https://mlsanity.com/paper/2207.09684/) (2022)
- [Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs](https://mlsanity.com/paper/2105.12049/) (2021)
- [Decoupling Local and Global Representations of Time Series](https://mlsanity.com/paper/2202.02262/) (2022)
- [Fair Representation: Guaranteeing Approximate Multiple Group Fairness for Unknown Tasks](https://mlsanity.com/paper/2109.00545/) (2021)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration](https://mlsanity.com/paper/2103.12300/) (2021)
- [Metadata Normalization](https://mlsanity.com/paper/2104.09052/) (2021)
- [Harmonization with Flow-based Causal Inference](https://mlsanity.com/paper/2106.06845/) (2021)
- [A Contrastive Objective for Learning Disentangled Representations](https://mlsanity.com/paper/2203.11284/) (2022)
- [Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Making](https://mlsanity.com/paper/2205.04790/) (2022)
- [FAIRER: Fairness as Decision Rationale Alignment](https://mlsanity.com/paper/2306.15299/) (2023)
- [Is Fairness Only Metric Deep? Evaluating and Addressing Subgroup Gaps in Deep Metric Learning](https://mlsanity.com/paper/2203.12748/) (2022)
- [Robust Concept Erasure via Kernelized Rate-Distortion Maximization](https://mlsanity.com/paper/2312.00194/) (2023)
- [Fairness-Aware Unsupervised Feature Selection](https://mlsanity.com/paper/2106.02216/) (2021)
- [Attributing Fair Decisions with Attention Interventions](https://mlsanity.com/paper/2109.03952/) (2021)
- [Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural Networks](https://mlsanity.com/paper/2204.04440/) (2022)
- [Toward Adversarial Training on Contextualized Language Representation](https://mlsanity.com/paper/2305.04557/) (2023)
- [Fair Bilevel Neural Network (FairBiNN): On Balancing fairness and accuracy via Stackelberg Equilibrium](https://mlsanity.com/paper/2410.16432/) (2024)

## Abstract (arXiv)

Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation. Unfortunately, learning representations that exhibit invariance to arbitrary nuisance factors yet remain useful for other tasks is challenging. Existing approaches cast the trade-off between task performance and invariance in an adversarial way, using an iterative minimax optimization. We show that adversarial training is unnecessary and sometimes counter-productive; we instead cast invariant representation learning as a single information-theoretic objective that can be directly optimized. We demonstrate that this approach matches or exceeds performance of state-of-the-art adversarial approaches for learning fair representations and for generative modeling with controllable transformations.