mlsanityData to 14 Oct 2026

Information Subtraction: Learning Representations for Conditional Entropy

arXiv 2501.02012 ↗First posted 2 Jan 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

22 of 35 references.

References · 22

  1. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  3. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  4. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 410
  5. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  6. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  7. 2015The Variational Fair Autoencodercited by 223
  8. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  9. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 122
  10. 2018Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddingscited by 114
  11. 2018Invariant Representations without Adversarial Trainingcited by 112
  12. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 99
10 more
  1. 2021Learning Temporally Causal Latent Processes from General Temporal Datacited by 98
  2. 2020Fundamental Limits and Tradeoffs in Invariant Representation Learningcited by 38
  3. 2019CCMI : Classifier based Conditional Mutual Information Estimationcited by 37
  4. 2021Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimizationcited by 32
  5. 2020Learning Unbiased Representations via Mutual Information Backpropagationcited by 25
  6. 2022Conditional Contrastive Learning with Kernelcited by 22
  7. 2020Representation Learning for Sequence Data with Deep Autoencoding Predictive Componentscited by 9
  8. 2022Disentangled Representation with Causal Constraints for Counterfactual Fairnesscited by 7
  9. 2019Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysiscited by 6
  10. 2020Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecastingcited by 5

Built on by

Not cited yet.