# A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification

arXiv: https://arxiv.org/abs/2310.13990 · HTML: https://mlsanity.com/paper/2310.13990/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 81 of its 120 references that are in the map.
- First posted on arXiv 21 Oct 2023.

## Builds on (most cited first, 30 of 81)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Towards Stable and Efficient Training of Verifiably Robust Neural Networks](https://mlsanity.com/paper/1906.06316/) (2019)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)

## Built on by (most cited first, 1 of 1)

- [Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism](https://mlsanity.com/paper/2402.12997/) (2024)

## Abstract (arXiv)

One of the pursued objectives of deep learning is to provide tools that learn abstract representations of reality from the observation of multiple contextual situations. More precisely, one wishes to extract disentangled representations which are (i) low dimensional and (ii) whose components are independent and correspond to concepts capturing the essence of the objects under consideration (Locatello et al., 2019b). One step towards this ambitious project consists in learning disentangled representations with respect to a predefined (sensitive) attribute, e.g., the gender or age of the writer. Perhaps one of the main application for such disentangled representations is fair classification. Existing methods extract the last layer of a neural network trained with a loss that is composed of a cross-entropy objective and a disentanglement regularizer. In this work, we adopt an information-theoretic view of this problem which motivates a novel family of regularizers that minimizes the mutual information between the latent representation and the sensitive attribute conditional to the target. The resulting set of losses, called CLINIC, is parameter free and thus, it is easier and faster to train. CLINIC losses are studied through extensive numerical experiments by training over 2k neural networks. We demonstrate that our methods offer a better disentanglement/accuracy trade-off than previous techniques, and generalize better than training with cross-entropy loss solely provided that the disentanglement task is not too constraining.