# Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization

arXiv: https://arxiv.org/abs/2108.05449 · HTML: https://mlsanity.com/paper/2108.05449/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 59 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification](https://mlsanity.com/paper/2001.01526/) (2020)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Federated Adversarial Domain Adaptation](https://mlsanity.com/paper/1911.02054/) (2019)
- [Learning from Failure: Training Debiased Classifier from Biased Classifier](https://mlsanity.com/paper/2007.02561/) (2020)
- [Learning Robust Representations by Projecting Superficial Statistics Out](https://mlsanity.com/paper/1903.06256/) (2019)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)

## Built on by (most cited first, 30 of 32)

- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Learning Debiased Classifier with Biased Committee](https://mlsanity.com/paper/2206.10843/) (2022)
- [Overcoming Simplicity Bias in Deep Networks using a Feature Sieve](https://mlsanity.com/paper/2301.13293/) (2023)
- [Generalizing Graph ODE for Learning Complex System Dynamics across Environments](https://mlsanity.com/paper/2307.04287/) (2023)
- [Localized Adversarial Domain Generalization](https://mlsanity.com/paper/2205.04114/) (2022)
- [On Fairness of Medical Image Classification with Multiple Sensitive Attributes via Learning Orthogonal Representations](https://mlsanity.com/paper/2301.01481/) (2023)
- [Pseudo Bias-Balanced Learning for Debiased Chest X-ray Classification](https://mlsanity.com/paper/2203.09860/) (2022)
- [Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment](https://mlsanity.com/paper/2111.13108/) (2021)
- [General Greedy De-bias Learning](https://mlsanity.com/paper/2112.10572/) (2021)
- [CAT: Controllable Attribute Translation for Fair Facial Attribute Classification](https://mlsanity.com/paper/2209.06850/) (2022)
- [Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization](https://mlsanity.com/paper/2409.03303/) (2024)
- [When Fairness Meets Privacy: Exploring Privacy Threats in Fair Binary Classifiers via Membership Inference Attacks](https://mlsanity.com/paper/2311.03865/) (2023)
- [Information-Theoretic Bounds on The Removal of Attribute-Specific Bias From Neural Networks](https://mlsanity.com/paper/2310.04955/) (2023)
- [Pushing the Accuracy-Group Robustness Frontier with Introspective Self-play](https://mlsanity.com/paper/2302.05807/) (2023)
- [Medical Image Debiasing by Learning Adaptive Agreement from a Biased Council](https://mlsanity.com/paper/2401.11713/) (2024)
- [Mitigating Spurious Correlations via Disagreement Probability](https://mlsanity.com/paper/2411.01757/) (2024)
- [Learning Disentangled Representations for Perceptual Point Cloud Quality Assessment via Mutual Information Minimization](https://mlsanity.com/paper/2411.07936/) (2024)
- [Do Fairness Interventions Come at the Cost of Privacy: Evaluations for Binary Classifiers](https://mlsanity.com/paper/2503.06150/) (2025)
- [Controllable Feature Whitening for Hyperparameter-Free Bias Mitigation](https://mlsanity.com/paper/2507.20284/) (2025)
- [Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information](https://mlsanity.com/paper/2603.03725/) (2026)
- [Mitigating Urban-Rural Disparities in Contrastive Representation Learning with Satellite Imagery](https://mlsanity.com/paper/2211.08672/) (2022)
- [Learning Fair Representations with High-Confidence Guarantees](https://mlsanity.com/paper/2310.15358/) (2023)
- [SABAF: Removing Strong Attribute Bias from Neural Networks with Adversarial Filtering](https://mlsanity.com/paper/2311.07141/) (2023)
- [Debiased Prompt Tuning in Vision-Language Model without Annotations](https://mlsanity.com/paper/2503.08368/) (2025)
- [Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains](https://mlsanity.com/paper/2509.00658/) (2025)
- [Delving into Identify-Emphasize Paradigm for Combating Unknown Bias](https://mlsanity.com/paper/2302.11414/) (2023)
- [Revisiting the Dataset Bias Problem from a Statistical Perspective](https://mlsanity.com/paper/2402.03577/) (2024)
- [From Discrete to Continuous: Deep Fair Clustering With Transferable Representations](https://mlsanity.com/paper/2403.16201/) (2024)
- [Information Subtraction: Learning Representations for Conditional Entropy](https://mlsanity.com/paper/2501.02012/) (2025)

## Abstract (arXiv)

Deep learning algorithms mine knowledge from the training data and thus would likely inherit the dataset's bias information. As a result, the obtained model would generalize poorly and even mislead the decision process in real-life applications. We propose to remove the bias information misused by the target task with a cross-sample adversarial debiasing (CSAD) method. CSAD explicitly extracts target and bias features disentangled from the latent representation generated by a feature extractor and then learns to discover and remove the correlation between the target and bias features. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. Moreover, we propose joint content and local structural representation learning to boost mutual information estimation for better performance. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.