# Learning Fair Representations via Rate-Distortion Maximization

arXiv: https://arxiv.org/abs/2202.00035 · HTML: https://mlsanity.com/paper/2202.00035/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 66 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [What's in a Name? Reducing Bias in Bios without Access to Protected Attributes](https://mlsanity.com/paper/1904.05233/) (2019)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings](https://mlsanity.com/paper/2007.00049/) (2020)
- [Evaluating Debiasing Techniques for Intersectional Biases](https://mlsanity.com/paper/2109.10441/) (2021)
- [A Rate-Distortion Framework for Explaining Neural Network Decisions](https://mlsanity.com/paper/1905.11092/) (2019)
- [Adversarial Scrubbing of Demographic Information for Text Classification](https://mlsanity.com/paper/2109.08613/) (2021)

## Built on by (most cited first, 15 of 15)

- [Exploring Safety-Utility Trade-Offs in Personalized Language Models](https://mlsanity.com/paper/2406.11107/) (2024)
- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion](https://mlsanity.com/paper/2311.07682/) (2023)
- [Robust Concept Erasure via Kernelized Rate-Distortion Maximization](https://mlsanity.com/paper/2312.00194/) (2023)
- [The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models](https://mlsanity.com/paper/2406.10130/) (2024)
- [Fundamental Limits of Perfect Concept Erasure](https://mlsanity.com/paper/2503.20098/) (2025)
- [Sustaining Fairness via Incremental Learning](https://mlsanity.com/paper/2208.12212/) (2022)
- [Debiasing Diffusion Model: Enhancing Fairness through Latent Representation Learning in Stable Diffusion Model](https://mlsanity.com/paper/2503.12536/) (2025)
- [Debiasing Multimodal Models via Causal Information Minimization](https://mlsanity.com/paper/2311.16941/) (2023)
- [Enhancing Group Fairness in Online Settings Using Oblique Decision Forests](https://mlsanity.com/paper/2310.11401/) (2023)
- [Nonlinear Concept Erasure: a Density Matching Approach](https://mlsanity.com/paper/2507.12341/) (2025)
- [Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translation](https://mlsanity.com/paper/2501.10150/) (2025)
- [Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure](https://mlsanity.com/paper/2603.07529/) (2026)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions](https://mlsanity.com/paper/2609.11253/) (2026)

## Abstract (arXiv)

Text representations learned by machine learning models often encode undesirable demographic information of the user. Predictive models based on these representations can rely on such information, resulting in biased decisions. We present a novel debiasing technique, Fairness-aware Rate Maximization (FaRM), that removes protected information by making representations of instances belonging to the same protected attribute class uncorrelated, using the rate-distortion function. FaRM is able to debias representations with or without a target task at hand. FaRM can also be adapted to remove information about multiple protected attributes simultaneously. Empirical evaluations show that FaRM achieves state-of-the-art performance on several datasets, and learned representations leak significantly less protected attribute information against an attack by a non-linear probing network.