mlsanityData to 7 Oct 2026

Gradient Matching for Domain Generalization

arXiv 2104.09937 ↗First posted 20 Apr 2021cs.LG stat.MLcited by 258

Not on the map. Why?

Abstract (arXiv)

Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive -- requires computation of second-order derivatives -- we derive a simpler first-order algorithm named Fish that approximates its optimization. We demonstrate the efficacy of Fish on 6 datasets from the Wilds benchmark, which captures distribution shift across a diverse range of modalities. Our method produces competitive results on these datasets and surpasses all baselines on 4 of them. We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). Builds on 36 of its 61 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

36 of 61 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2019Invariant Risk Minimizationcited by 1,832
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  10. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  12. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
24 more
  1. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  2. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  3. 2020In Search of Lost Domain Generalizationcited by 1,042
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  5. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  6. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  7. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  8. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  9. 2018Recognition in Terra Incognitacited by 655
  10. 2016Learning to learn by gradient descent by gradient descentcited by 616
  11. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 449
  12. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  13. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  14. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  15. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 322
  16. 2020Learning to Generate Novel Domains for Domain Generalizationcited by 296
  17. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  18. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  19. 2019Episodic Training for Domain Generalizationcited by 245
  20. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  21. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  22. 2020Learning explanations that are hard to varycited by 132
  23. 2020Sequential Learning for Domain Generalizationcited by 25
  24. 2017ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biasescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 56; 2023: 75; 2024: 57; 2025: 31; 2026: 26.

258 papers.

Sort:

By area:

  1. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  2. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  3. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  4. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  5. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  6. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  7. 2021OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalizationcited by 86
  8. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  9. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  10. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  11. 2022C-Mixup: Improving Generalization in Regressioncited by 57
  12. 2021SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalizationcited by 50
38 more
  1. 2023AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptationcited by 48
  2. 2023Improved Test-Time Adaptation for Domain Generalizationcited by 45
  3. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  4. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  5. 2022ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networkscited by 39
  6. 2023Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimizationcited by 39
  7. 2022Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalizationcited by 38
  8. 2022PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalizationcited by 38
  9. 2022Invariant and Transportable Representations for Anti-Causal Domain Shiftscited by 37
  10. 2022Learning Domain Invariant Prompt for Vision-Language Modelscited by 35
  11. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  12. 2022Rich Feature Construction for the Optimization-Generalization Dilemmacited by 34
  13. 2022UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixupcited by 34
  14. 2022Domain Generalization using Pretrained Models without Fine-tuningcited by 29
  15. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  16. 2022Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognitioncited by 28
  17. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  18. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  19. 2022On Multi-Domain Long-Tailed Recognition, Imbalanced Domain Generalization and Beyondcited by 26
  20. 2024Gradient Alignment for Cross-Domain Face Anti-Spoofingcited by 26
  21. 2023Federated Domain Generalization: A Surveycited by 25
  22. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  23. 2022Learning Invariant Visual Representations for Compositional Zero-Shot Learningcited by 24
  24. 2022How Robust is Unsupervised Representation Learning to Distribution Shift?cited by 24
  25. 2023Fairness and Accuracy under Domain Generalizationcited by 24
  26. 2023Generalizable Decision Boundaries: Dualistic Meta-Learning for Open Set Domain Generalizationcited by 24
  27. 2023Distribution Shift Inversion for Out-of-Distribution Predictioncited by 23
  28. 2024Rethinking Multi-domain Generalization with A General Learning Objectivecited by 23
  29. 2023MADG: Margin-based Adversarial Learning for Domain Generalizationcited by 22
  30. 2024HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shiftscited by 22
  31. 2022Domain-Unified Prompt Representations for Source-Free Domain Generalizationcited by 21
  32. 2023Spuriosity Didn't Kill the Classifier: Using Invariant Predictions to Harness Spurious Featurescited by 21
  33. 2023DomainDrop: Suppressing Domain-Sensitive Channels for Domain Generalizationcited by 21
  34. 2021MVDG: A Unified Multi-view Framework for Domain Generalizationcited by 20
  35. 2022Towards Robust Adaptive Object Detection under Noisy Annotationscited by 20
  36. 2022Causal Balancing for Domain Generalizationcited by 20
  37. 2022Meta Spatio-Temporal Debiasing for Video Scene Graph Generationcited by 20
  38. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20

50 most cited of 258 All 258 (JSON)