How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis
Not on the map. Why?
Abstract (arXiv)
Domain generalization aims to learn invariance across multiple training domains, thereby enhancing generalization against out-of-distribution data. While gradient or representation matching algorithms have achieved remarkable success, these methods generally lack generalization guarantees or depend on strong assumptions, leaving a gap in understanding the underlying mechanism of distribution matching. In this work, we formulate domain generalization from a novel probabilistic perspective, ensuring robustness while avoiding overly conservative solutions. Through comprehensive information-theoretic analysis, we provide key insights into the roles of gradient and representation matching in promoting generalization. Our results reveal the complementary relationship between these two components, indicating that existing works focusing solely on either gradient or representation alignment are insufficient to solve the domain generalization problem. In light of these theoretical findings, we introduce IDM to simultaneously align the inter-domain gradients and representations. Integrated with the proposed PDM method for complex distribution matching, IDM achieves superior performance over various baseline methods.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 69 of its 88 references that are in the map. First posted on arXiv 14 Jun 2024.
Builds on
69 of 88 references.
References · 69
- 2015Domain-Adversarial Training of Neural Networkscited by 3,023
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2019Invariant Risk Minimizationcited by 1,832
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
- 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
- 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
- 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
- 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
- 2020In Search of Lost Domain Generalizationcited by 1,042
- 2017Deeper, Broader and Artier Domain Generalizationcited by 983
- 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
38 more
- 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
- 2018Recognition in Terra Incognitacited by 655
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
- 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
- 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
- 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
- 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
- 2021Gradient Matching for Domain Generalizationcited by 258
- 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
- 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
- 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
- 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
- 2021SelfReg: Self-supervised Contrastive Regularization for Domain Generalizationcited by 189
- 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
- 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
- 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
- 2017Domain Generalization by Marginal Transfer Learningcited by 180
- 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
- 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
- 2020Learning explanations that are hard to varycited by 132
- 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
- 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
- 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
- 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
- 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
- 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
- 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 79
- 2020The Conditional Entropy Bottleneckcited by 75
- 2020Learning Invariant Representations and Risks for Semi-supervised Domain Adaptationcited by 69
- 2020A causal framework for distribution generalizationcited by 68
- 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
- 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
- 2023Generalization Bounds: Perspectives from Information Theory and PAC-Bayescited by 60
- 2020When is invariance useful in an Out-of-Distribution Generalization problem ?cited by 52
- 2021SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalizationcited by 50
- 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
- 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
- 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
50 most cited of 69 All 69 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.