mlsanityData to 14 Oct 2026

Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts

arXiv 2405.18861 ↗First posted 29 May 2024cs.CV cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents a Domain-Inspired Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts. It is motivated by the inconsistent convergence degree of SAM across different domains, which induces optimization bias towards certain domains and thus impairs the overall convergence. To address this issue, we consider the domain-level convergence consistency in the sharpness estimation to prevent the overwhelming (deficient) perturbations for less (well) optimized domains. Specifically, DISAM introduces the constraint of minimizing variance in the domain loss, which allows the elastic gradient calibration in perturbation generation: when one domain is optimized above the averaging level textitw.r.t. loss, the gradient perturbation towards that domain will be weakened automatically, and vice versa. Under this mechanism, we theoretically show that DISAM can achieve faster overall convergence and improved generalization in principle when inconsistent convergence emerges. Extensive experiments on various domain generalization benchmarks show the superiority of DISAM over a range of state-of-the-art methods. Furthermore, we show the superior efficiency of DISAM in parameter-efficient fine-tuning combined with the pretraining models. The source code is released at https://github.com/MediaBrain-SJTU/DISAM.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

54 of 72 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2019Invariant Risk Minimizationcited by 1,854
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  11. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  12. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
38 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  2. 2020In Search of Lost Domain Generalizationcited by 1,054
  3. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  4. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  5. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
  6. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  7. 2018Recognition in Terra Incognitacited by 658
  8. 2021Domain Generalization with MixStylecited by 566
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  11. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  12. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 407
  13. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  14. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 323
  15. 2020Learning to Generate Novel Domains for Domain Generalizationcited by 296
  16. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  17. 2021A Fourier-based Framework for Domain Generalizationcited by 290
  18. 2021Gradient Matching for Domain Generalizationcited by 258
  19. 2020Deep Domain-Adversarial Image Generation for Domain Generalisationcited by 231
  20. 2019Domain-Specific Batch Normalization for Unsupervised Domain Adaptationcited by 197
  21. 2021SelfReg: Self-supervised Contrastive Regularization for Domain Generalizationcited by 189
  22. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  23. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  24. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  25. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 161
  26. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  27. 2019Feature-Critic Networks for Heterogeneous Domain Generalizationcited by 130
  28. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  29. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  30. 2020Learning from Extrinsic and Intrinsic Supervisions for Domain Generalizationcited by 108
  31. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  32. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  33. 2022Sharpness-Aware Training for Freecited by 95
  34. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  35. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  36. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  37. 2021Style Normalization and Restitution for Domain Generalization and Adaptationcited by 46
  38. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 5.

12 papers.

  1. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  2. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 10
  3. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  4. 2025Noise-Aware Generalization: Robustness to In-Domain Noise and Out-of-Domain Generalizationcited by 2
  5. 2025Flatness Guided Test-Time Adaptation for Vision-Language Modelscited by 1
  6. 2026One-Step Generalization Ratio Guided Optimization for Domain Generalizationcited by 1
  7. 2026Flat-Consensus Diffusion for Robust Data Reshaping under Noisy Evaluatorcited by 1
  8. 2025DGSAM: Domain Generalization via Individual Sharpness-Aware Minimizationcited by 0
  9. 2025Learning to Generate Cross-Task Unexploitable Examplescited by 0
  10. 2026Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learningcited by 0
  11. 2026TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillationcited by 0
  12. 2026FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filteringcited by 0