# On Characterizing the Trade-off in Invariant Representation Learning

arXiv: https://arxiv.org/abs/2109.03386 · HTML: https://mlsanity.com/paper/2109.03386/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 66 references that are in the map.
- First posted on arXiv 8 Sep 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Censoring Representations with an Adversary](https://mlsanity.com/paper/1511.05897/) (2015)
- [Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations](https://mlsanity.com/paper/1707.00075/) (2017)
- [Minimax Pareto Fairness: A Multi Objective Perspective](https://mlsanity.com/paper/2011.01821/) (2020)
- [On the Fairness of Disentangled Representations](https://mlsanity.com/paper/1905.13662/) (2019)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Learning Controllable Fair Representations](https://mlsanity.com/paper/1812.04218/) (2018)
- [Self-Supervised Learning with Kernel Dependence Maximization](https://mlsanity.com/paper/2106.08320/) (2021)
- [Privacy-preserving Neural Representations of Text](https://mlsanity.com/paper/1808.09408/) (2018)
- [Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach](https://mlsanity.com/paper/1904.05514/) (2019)
- [Fair Kernel Learning](https://mlsanity.com/paper/1710.05578/) (2017)
- [Fundamental Limits and Tradeoffs in Invariant Representation Learning](https://mlsanity.com/paper/2012.10713/) (2020)
- [Adversarial Learning of Privacy-Preserving and Task-Oriented Representations](https://mlsanity.com/paper/1911.10143/) (2019)
- [Learning Unbiased Representations via R'enyi Minimization](https://mlsanity.com/paper/2009.03183/) (2020)
- [Minimax Filter: Learning to Preserve Privacy from Inference Attacks](https://mlsanity.com/paper/1610.03577/) (2016)
- [On the Global Optima of Kernelized Adversarial Representation Learning](https://mlsanity.com/paper/1910.07423/) (2019)
- [Privacy-Preserving Image Features via Adversarial Affine Subspace Embeddings](https://mlsanity.com/paper/2006.06634/) (2020)

## Built on by (most cited first, 12 of 12)

- [Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalization](https://mlsanity.com/paper/2206.07766/) (2022)
- [FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs](https://mlsanity.com/paper/2403.15593/) (2024)
- [Utility-Fairness Trade-Offs and How to Find Them](https://mlsanity.com/paper/2404.09454/) (2024)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes](https://mlsanity.com/paper/2501.00962/) (2025)
- [Fundamental Limits of Perfect Concept Erasure](https://mlsanity.com/paper/2503.20098/) (2025)
- [Minority Collective Action for User-Side Fairness](https://mlsanity.com/paper/2508.15374/) (2025)
- [Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure](https://mlsanity.com/paper/2603.07529/) (2026)
- [Fair Domain Generalization: An Information-Theoretic View](https://mlsanity.com/paper/2507.05823/) (2025)
- [PolyJuice Makes It Real: Black-Box, Universal Red Teaming for Synthetic Image Detectors](https://mlsanity.com/paper/2509.15551/) (2025)
- [Deep Fair Learning: Task-Aware Fair Representations via Joint Distance-Covariance Regularization](https://mlsanity.com/paper/2504.06470/) (2025)
- [FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectors](https://mlsanity.com/paper/2606.05101/) (2026)

## Abstract (arXiv)

Many applications of representation learning, such as privacy preservation, algorithmic fairness, and domain adaptation, desire explicit control over semantic information being discarded. This goal is formulated as satisfying two objectives: maximizing utility for predicting a target attribute while simultaneously being invariant (independent) to a known semantic attribute. Solutions to invariant representation learning (IRepL) problems lead to a trade-off between utility and invariance when they are competing. While existing works study bounds on this trade-off, two questions remain outstanding: 1) What is the exact trade-off between utility and invariance? and 2) What are the encoders (mapping the data to a representation) that achieve the trade-off, and how can we estimate it from training data? This paper addresses these questions for IRepLs in reproducing kernel Hilbert spaces (RKHS)s. Under the assumption that the distribution of a low-dimensional projection of high-dimensional data is approximately normal, we derive a closed-form solution for the global optima of the underlying optimization problem for encoders in RKHSs. This yields closed formulae for a near-optimal trade-off, corresponding optimal representation dimensionality, and the corresponding encoder(s). We also numerically quantify the trade-off on representative problems and compare them to those achieved by baseline IRepL algorithms.