# Information Removal at the bottleneck in Deep Neural Networks

arXiv: https://arxiv.org/abs/2210.00891 · HTML: https://mlsanity.com/paper/2210.00891/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Machine Learning Models that Remember Too Much](https://mlsanity.com/paper/1709.07886/) (2017)
- [FairGAN: Fairness-aware Generative Adversarial Networks](https://mlsanity.com/paper/1805.11202/) (2018)
- [Learning from Failure: Training Debiased Classifier from Biased Classifier](https://mlsanity.com/paper/2007.02561/) (2020)
- [EnD: Entangling and Disentangling deep representations for bias correction](https://mlsanity.com/paper/2103.02023/) (2021)
- [Neural Network with Unbounded Activation Functions is Universal Approximator](https://mlsanity.com/paper/1505.03654/) (2015)
- [Fairness GAN](https://mlsanity.com/paper/1805.09910/) (2018)
- [A Hybrid Deep Learning Architecture for Privacy-Preserving Mobile Analytics](https://mlsanity.com/paper/1703.02952/) (2017)
- [UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models](https://mlsanity.com/paper/2110.15499/) (2021)
- [Measuring the Biases and Effectiveness of Content-Style Disentanglement](https://mlsanity.com/paper/2008.12378/) (2020)
- [Feature and Label Embedding Spaces Matter in Addressing Image Classifier Bias](https://mlsanity.com/paper/2110.14336/) (2021)
- [Everybody Is Unique: Towards Unbiased Human Mesh Recovery](https://mlsanity.com/paper/2107.06239/) (2021)

## Built on by (most cited first, 4 of 4)

- [Mining bias-target Alignment from Voronoi Cells](https://mlsanity.com/paper/2305.03691/) (2023)
- [Debiasing surgeon: fantastic weights and how to find them](https://mlsanity.com/paper/2403.14200/) (2024)
- [Bias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Models](https://mlsanity.com/paper/2603.05582/) (2026)
- [CutClean: Neural Network Pruning for Privacy-Preserving Inference](https://mlsanity.com/paper/2608.13773/) (2026)

## Abstract (arXiv)

Deep learning models are nowadays broadly deployed to solve an incredibly large variety of tasks. Commonly, leveraging over the availability of "big data", deep neural networks are trained as black-boxes, minimizing an objective function at its output. This however does not allow control over the propagation of some specific features through the model, like gender or race, for solving some an uncorrelated task. This raises issues either in the privacy domain (considering the propagation of unwanted information) and of bias (considering that these features are potentially used to solve the given task). In this work we propose IRENE, a method to achieve information removal at the bottleneck of deep neural networks, which explicitly minimizes the estimated mutual information between the features to be kept ``private'' and the target. Experiments on a synthetic dataset and on CelebA validate the effectiveness of the proposed approach, and open the road towards the development of approaches guaranteeing information removal in deep neural networks.