mlsanityData to 7 Oct 2026

SWAD: Domain Generalization by Seeking Flat Minima

arXiv 2102.08604 ↗First posted 17 Feb 2021cs.LG cs.CVcited by 431

Not on the map. Why?

Abstract (arXiv)

Domain generalization (DG) methods aim to achieve generalizability to an unseen target domain by using only training data from the source domains. Although a variety of DG methods have been proposed, a recent study shows that under a fair evaluation protocol, called DomainBed, the simple empirical risk minimization (ERM) approach works comparable to or even outperforms previous methods. Unfortunately, simply solving ERM on a complex, non-convex loss function can easily lead to sub-optimal generalizability by seeking sharp minima. In this paper, we theoretically show that finding flat minima results in a smaller domain generalization gap. We also propose a simple yet effective method, named Stochastic Weight Averaging Densely (SWAD), to find flat minima. SWAD finds flatter minima and suffers less from overfitting than does the vanilla SWA by a dense and overfit-aware stochastic weight sampling strategy. SWAD shows state-of-the-art performances on five DG benchmarks, namely PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet, with consistent and large margins of +1.6% averagely on out-of-domain accuracy. We also compare SWAD with conventional generalization methods, such as data augmentation and consistency regularization methods, to verify that the remarkable performance improvements are originated from by seeking flat minima, not from better in-domain generalizability. Last but not least, SWAD is readily adaptable to existing DG methods without modification; the combination of SWAD and an existing DG method further improves DG performances. Source code is available at https://github.com/khanrc/swad.

Key facts. Cited by 431 papers in the map (data through 7 Oct 2026). Builds on 52 of its 73 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

52 of 73 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2019Invariant Risk Minimizationcited by 1,832
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  6. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  7. 2020In Search of Lost Domain Generalizationcited by 1,042
  8. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  9. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  10. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  11. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  12. 2018Recognition in Terra Incognitacited by 655
  13. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  14. 2013Domain Generalization via Invariant Feature Representationcited by 622
  15. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  16. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  17. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  18. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 449
  19. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  20. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  21. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  22. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  23. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  24. 2018Generalizing Across Domains via Cross-Gradient Trainingcited by 268
  25. 2019Adversarial Domain Adaptation with Domain Mixupcited by 250
  26. 2019Learning De-biased Representations with Biased Representationscited by 199
  27. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  28. 2017Domain Generalization by Marginal Transfer Learningcited by 180
  29. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 160
  30. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  31. 2019Does Object Recognition Work for Everyone?cited by 149
  32. 2018Domain Generalization via Conditional Invariant Representationcited by 146
  33. 2020Learning to Balance Specificity and Invariance for In and Out of Domain Generalizationcited by 128
  34. 2018Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttimecited by 122
  35. 2020Learning from Extrinsic and Intrinsic Supervisions for Domain Generalizationcited by 108
  36. 2020Heterogeneous Domain Generalization via Domain Mixupcited by 103
  37. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  38. 2020Permuted AdaIN: Reducing the Bias Towards Global Statistics in Image Classificationcited by 63

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 55; 2023: 117; 2024: 113; 2025: 92; 2026: 40.

431 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  4. 2021Domain Generalization: A Surveycited by 760
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  6. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  7. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  8. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  9. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  11. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  12. 2023Mitigating the Alignment Tax of RLHFcited by 181
38 more
  1. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  2. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  3. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  4. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  5. 2024Knowledge Fusion of Large Language Modelscited by 110
  6. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  7. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  8. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  9. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  10. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  11. 2023Deep Model Fusion: A Surveycited by 89
  12. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  13. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  14. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  15. 2022A Closer Look at Smoothness in Domain Adversarial Trainingcited by 85
  16. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  17. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  18. 2022When Do Flat Minima Optimizers Work?cited by 78
  19. 2022MEDFAIR: Benchmarking Fairness for Medical Imagingcited by 66
  20. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  21. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  22. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  23. 2022Prompt Vision Transformer for Domain Generalizationcited by 52
  24. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  25. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  26. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  27. 2023Feature Alignment and Uniformity for Test Time Adaptationcited by 42
  28. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  29. 2021Federated Learning with Domain Generalizationcited by 41
  30. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  31. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  32. 2021Learning Fair Classifiers with Partially Annotated Group Labelscited by 40
  33. 2023Improved Probabilistic Image-Text Representationscited by 40
  34. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  35. 2021Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domainscited by 39
  36. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  37. 2022PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalizationcited by 38
  38. 2023StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalizationcited by 36

50 most cited of 431 All 431 (JSON)