Disentangled Representation Learning with Transmitted Information Bottleneck
Not on the map. Why?
Abstract (arXiv)
Encoding only the task-related information from the raw data, ie, disentangled representation learning, can greatly contribute to the robustness and generalizability of models. Although significant advances have been made by regularizing the information in representations with information theory, two major challenges remain: 1) the representation compression inevitably leads to performance drop; 2) the disentanglement constraints on representations are in complicated optimization. To these issues, we introduce Bayesian networks with transmitted information to formulate the interaction among input and representations during disentanglement. Building upon this framework, we propose DisTIB (Transmitted Information Bottleneck for Disentangled representation learning), a novel objective that navigates the balance between information compression and preservation. We employ variational inference to derive a tractable estimation for DisTIB. This estimation can be simply optimized via standard gradient descent with a reparameterization trick. Moreover, we theoretically prove that DisTIB can achieve optimal disentanglement, underscoring its superior efficacy. To solidify our claims, we conduct extensive experiments on various downstream tasks to demonstrate the appealing efficacy of DisTIB and validate our theoretical analyses.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 62 of its 103 references that are in the map. First posted on arXiv 3 Nov 2023.
Builds on
62 of 103 references.
References · 62
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2013Intriguing properties of neural networkscited by 5,114
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2017Prototypical Networks for Few-shot Learningcited by 3,001
- 2016Matching Networks for One Shot Learningcited by 2,246
- 2016Deep Variational Information Bottleneckcited by 915
- 2018Robustness May Be at Odds with Accuracycited by 873
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
- 2015Deep Learning and the Information Bottleneck Principlecited by 793
- 2018Recognition in Terra Incognitacited by 655
- 2018Disentangling by Factorisingcited by 614
- 2017Generative Adversarial Networks: An Overviewcited by 588
38 more
- 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
- 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
- 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
- 2020The Risks of Invariant Risk Minimizationcited by 262
- 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
- 2020Graph Information Bottleneckcited by 168
- 2020Few-Shot Classification with Feature Map Reconstruction Networkscited by 116
- 2020Interventional Few-Shot Learningcited by 112
- 2020Learning to Learn with Variational Information Bottleneck for Domain Generalizationcited by 102
- 2021Counterfactual Zero-Shot and Open-Set Visual Recognitioncited by 95
- 2016Disentangling factors of variation in deep representations using adversarial trainingcited by 87
- 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
- 2021Relational Embedding for Few-Shot Classificationcited by 74
- 2017Nonlinear Information Bottleneckcited by 71
- 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
- 2022High-resolution Face Swapping via Latent Semantics Disentanglementcited by 57
- 2021Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learningcited by 56
- 2019Learning Disentangled Representations via Mutual Information Estimationcited by 52
- 2020Prototype Completion with Primitive Knowledge for Few-Shot Learningcited by 50
- 2022Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classificationcited by 50
- 2020DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learningcited by 49
- 2021Learning Dynamic Alignment via Meta-filter for Few-shot Learningcited by 41
- 2022Generating Representative Samples for Few-Shot Classificationcited by 39
- 2022Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentationcited by 37
- 2021Partner-Assisted Learning for Few-Shot Image Classificationcited by 36
- 2018Caveats for information bottleneck in deterministic scenarioscited by 34
- 2022Few-Shot Classification with Contrastive Learningcited by 31
- 2020Variational Feature Disentangling for Fine-Grained Few-Shot Classificationcited by 26
- 2021Binocular Mutual Learning for Improving Few-shot Classificationcited by 26
- 2020Disentangled Information Bottleneckcited by 24
- 2020Variational Interaction Information Maximization for Cross-domain Disentanglementcited by 23
- 2022Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learningcited by 22
- 2017Group invariance principles for causal generative modelscited by 20
- 2022Multi-Behavior Sequential Recommendation with Temporal Graph Transformercited by 20
- 2020Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximizationcited by 19
- 2021Learning to Affiliate: Mutual Centralized Learning for Few-shot Classificationcited by 18
- 2022When Does Group Invariant Learning Survive Spurious Correlations?cited by 18
- 2013Multivariate Information Bottleneckcited by 17
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.