mlsanityData to 14 Oct 2026

Scalable Infomin Learning

arXiv 2302.10701 ↗First posted 21 Feb 2023cs.LG cs.AI cs.ITcited by 8

Not on the map. Why?

Abstract (arXiv)

The task of infomin learning aims to learn a representation with high utility while being uninformative about a specified target, with the latter achieved by minimising the mutual information between the representation and the target. It has broad applications, ranging from training fair prediction models against protected attributes, to unsupervised learning with disentangled representations. Recent works on infomin learning mainly use adversarial training, which involves training a neural network to estimate mutual information or its proxy and thus is slow and difficult to optimise. Drawing on recent advances in slicing techniques, we propose a new infomin learning approach, which uses a novel proxy metric to mutual information. We further derive an accurate and analytically computable approximation to this proxy metric, thereby removing the need of constructing neural network-based mutual information estimators. Experiments on algorithmic fairness, disentangled representation learning and domain adaptation verify that our method can effectively remove unwanted information with limited time budget.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 48 of its 72 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

48 of 72 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  8. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  9. 2016Deep Variational Information Bottleneckcited by 930
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  11. 2018MINE: Mutual Information Neural Estimationcited by 683
  12. 2018Disentangling by Factorisingcited by 616
36 more
  1. 2019On Variational Bounds of Mutual Informationcited by 559
  2. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  3. 2018Learning Adversarially Fair and Transferable Representationscited by 353
  4. 2018Understanding disentangling in β-VAEcited by 335
  5. 2018Towards a Definition of Disentangled Representationscited by 296
  6. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 241
  7. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  8. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  9. 2019Flexibly Fair Representation Learning by Disentanglementcited by 196
  10. 2015Censoring Representations with an Adversarycited by 192
  11. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 170
  12. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  13. 2019Wasserstein Fair Classificationcited by 131
  14. 2017Controllable Invariance through Adversarial Feature Learningcited by 127
  15. 2018Invariant Representations without Adversarial Trainingcited by 112
  16. 2018Generative Modeling using the Sliced Wasserstein Distancecited by 108
  17. 2020Fair Regression with Wasserstein Barycenterscited by 100
  18. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  19. 2020Statistical and Topological Properties of Sliced Probability Divergencescited by 85
  20. 2018Learning Controllable Fair Representationscited by 83
  21. 2019Overlearning Reveals Sensitive Attributescited by 76
  22. 2020Neural Approximate Sufficient Statistics for Implicit Modelscited by 74
  23. 2019Mitigating Information Leakage in Image Representations: A Maximum Entropy Approachcited by 48
  24. 2021Fair Normalizing Flowscited by 38
  25. 2019Orthogonal Estimation of Wasserstein Distancescited by 32
  26. 2019Learning Fair Representations via an Adversarial Frameworkcited by 29
  27. 2021Commutative Lie Group VAE for Disentanglement Learningcited by 27
  28. 2013The Randomized Dependence Coefficientcited by 26
  29. 2018CR-GAN: Learning Complete Representations for Multi-view Generationcited by 23
  30. 2020Sliced Kernelized Stein Discrepancycited by 23
  31. 2014Discovering Hidden Factors of Variation in Deep Networkscited by 21
  32. 2013Partial Distance Correlation with Methods for Dissimilaritiescited by 20
  33. 2019Invariant Representations through Adversarial Forgettingcited by 20
  34. 2019Fairness-Aware Neural R'eyni Minimization for Continuous Featurescited by 15
  35. 2022Controlling Directions Orthogonal to a Classifiercited by 13
  36. 2022Disentangled Representation Learning for RF Fingerprint Extraction under Unknown Channel Statisticscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 1; 2026: 1.

8 papers.

  1. 2024Approximating mutual information of high-dimensional variables using learned representationscited by 21
  2. 2023Max-Sliced Mutual Informationcited by 10
  3. 2025Neural Mutual Information Estimation with Vector Copulascited by 4
  4. 2023Towards Poisoning Fair Representationscited by 3
  5. 2026InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimatecited by 1
  6. 2024From Discrete to Continuous: Deep Fair Clustering With Transferable Representationscited by 0
  7. 2024Generalizing Orthogonalization for Models with Non-Linearitiescited by 0
  8. 2024Mutual Information Multinomial Estimationcited by 0