# Learning Fair Representation via Distributional Contrastive Disentanglement

arXiv: https://arxiv.org/abs/2206.08743 · HTML: https://mlsanity.com/paper/2206.08743/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Adversarial Domain Adaptation with Domain Mixup](https://mlsanity.com/paper/1912.01805/) (2019)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Large-Scale Methods for Distributionally Robust Optimization](https://mlsanity.com/paper/2010.05893/) (2020)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Disentangling factors of variation in deep representations using adversarial training](https://mlsanity.com/paper/1611.03383/) (2016)
- [Fairness by Learning Orthogonal Disentangled Representations](https://mlsanity.com/paper/2003.05707/) (2020)
- [FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders](https://mlsanity.com/paper/2103.06413/) (2021)
- [BiaSwap: Removing dataset bias with bias-tailored swapping augmentation](https://mlsanity.com/paper/2108.10008/) (2021)
- [Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach](https://mlsanity.com/paper/1904.05514/) (2019)
- [Counterfactual Fairness with Disentangled Causal Effect Variational Autoencoder](https://mlsanity.com/paper/2011.11878/) (2020)
- [Moment Matching Deep Contrastive Latent Variable Models](https://mlsanity.com/paper/2202.10560/) (2022)

## Built on by (most cited first, 30 of 31)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes](https://mlsanity.com/paper/2402.01981/) (2024)
- [FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering](https://mlsanity.com/paper/2504.14492/) (2025)
- [Supervised Algorithmic Fairness in Distribution Shifts: A Survey](https://mlsanity.com/paper/2402.01327/) (2024)
- [Efficient fair PCA for fair representation learning](https://mlsanity.com/paper/2302.13319/) (2023)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [FARE: Provably Fair Representation Learning with Practical Certificates](https://mlsanity.com/paper/2210.07213/) (2022)
- [Domain-Specific Risk Minimization for Out-of-Distribution Generalization](https://mlsanity.com/paper/2208.08661/) (2022)
- [Towards Fair Disentangled Online Learning for Changing Environments](https://mlsanity.com/paper/2306.01007/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Algorithmic Fairness Generalization under Covariate and Dependence Shifts Simultaneously](https://mlsanity.com/paper/2311.13816/) (2023)
- [Learning Fair Invariant Representations under Covariate and Correlation Shifts Simultaneously](https://mlsanity.com/paper/2408.09312/) (2024)
- [Addressing Bias in Generative AI: Challenges and Research Opportunities in Information Management](https://mlsanity.com/paper/2502.10407/) (2025)
- [Fair Representation Learning for Continuous Sensitive Attributes using Expectation of Integral Probability Metrics](https://mlsanity.com/paper/2505.06435/) (2025)
- [Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents](https://mlsanity.com/paper/2606.26080/) (2026)
- [Privacy-Preserved Neural Graph Databases](https://mlsanity.com/paper/2312.15591/) (2023)
- [Achieving Fairness Across Local and Global Models in Federated Learning](https://mlsanity.com/paper/2406.17102/) (2024)
- [Debiasing Diffusion Model: Enhancing Fairness through Latent Representation Learning in Stable Diffusion Model](https://mlsanity.com/paper/2503.12536/) (2025)
- [Independent Distribution Regularization for Private Graph Embedding](https://mlsanity.com/paper/2308.08360/) (2023)
- [FADE: Towards Fairness-aware Generation for Domain Generalization via Classifier-Guided Score-based Diffusion Models](https://mlsanity.com/paper/2406.09495/) (2024)
- [Enhancing Fairness in Unsupervised Graph Anomaly Detection through Disentanglement](https://mlsanity.com/paper/2406.00987/) (2024)
- [FEED: Fairness-Enhanced Meta-Learning for Domain Generalization](https://mlsanity.com/paper/2411.01316/) (2024)
- [Learning Fair Representations with High-Confidence Guarantees](https://mlsanity.com/paper/2310.15358/) (2023)
- [Debiasing Graph Representation Learning based on Information Bottleneck](https://mlsanity.com/paper/2409.01367/) (2024)
- [Fairness-aware Anomaly Detection via Fair Projection](https://mlsanity.com/paper/2505.11132/) (2025)
- [Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains](https://mlsanity.com/paper/2509.00658/) (2025)
- [Nearest-Neighbor Density Estimation for Dependency Suppression](https://mlsanity.com/paper/2603.04224/) (2026)

## Abstract (arXiv)

Learning fair representation is crucial for achieving fairness or debiasing sensitive information. Most existing works rely on adversarial representation learning to inject some invariance into representation. However, adversarial learning methods are known to suffer from relatively unstable training, and this might harm the balance between fairness and predictiveness of representation. We propose a new approach, learning FAir Representation via distributional CONtrastive Variational AutoEncoder (FarconVAE), which induces the latent space to be disentangled into sensitive and nonsensitive parts. We first construct the pair of observations with different sensitive attributes but with the same labels. Then, FarconVAE enforces each non-sensitive latent to be closer, while sensitive latents to be far from each other and also far from the non-sensitive latent by contrasting their distributions. We provide a new type of contrastive loss motivated by Gaussian and Student-t kernels for distributional contrastive learning with theoretical analysis. Besides, we adopt a new swap-reconstruction loss to boost the disentanglement further. FarconVAE shows superior performance on fairness, pretrained model debiasing, and domain generalization tasks from various modalities, including tabular, image, and text.