mlsanityData to 14 Oct 2026

Flatness-Aware Minimization for Domain Generalization

arXiv 2307.11108 ↗First posted 20 Jul 2023cs.CV cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Domain generalization (DG) seeks to learn robust models that generalize well under unknown distribution shifts. As a critical aspect of DG, optimizer selection has not been explored in depth. Currently, most DG methods follow the widely used benchmark, DomainBed, and utilize Adam as the default optimizer for all datasets. However, we reveal that Adam is not necessarily the optimal choice for the majority of current DG methods and datasets. Based on the perspective of loss landscape flatness, we propose a novel approach, Flatness-Aware Minimization for Domain Generalization (FAD), which can efficiently optimize both zeroth-order and first-order flatness simultaneously for DG. We provide theoretical analyses of the FAD's out-of-distribution (OOD) generalization error and convergence. Our experimental results demonstrate the superiority of FAD on various DG datasets. Additionally, we confirm that FAD is capable of discovering flatter optima in comparison to other zeroth-order and first-order flatness-aware optimization methods.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 95 of its 130 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

95 of 130 references.

References · 95

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  6. 2019Invariant Risk Minimizationcited by 1,854
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  12. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
38 more
  1. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  2. 2020In Search of Lost Domain Generalizationcited by 1,054
  3. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  4. 2020Designing Network Design Spacescited by 960
  5. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  6. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  7. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
  8. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  9. 2018Recognition in Terra Incognitacited by 658
  10. 2013Domain Generalization via Invariant Feature Representationcited by 623
  11. 2021Domain Generalization with MixStylecited by 566
  12. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  13. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  14. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 450
  15. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  16. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  17. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  18. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 407
  19. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  20. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 323
  21. 2020Learning to Learn Single Domain Generalizationcited by 296
  22. 2020Learning to Generate Novel Domains for Domain Generalizationcited by 296
  23. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  24. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  25. 2021A Fourier-based Framework for Domain Generalizationcited by 290
  26. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  27. 2017Three Factors Influencing Minima in SGDcited by 273
  28. 2018Generalizing Across Domains via Cross-Gradient Trainingcited by 268
  29. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  30. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  31. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  32. 2019Adversarial Domain Adaptation with Domain Mixupcited by 250
  33. 2019Episodic Training for Domain Generalizationcited by 245
  34. 2020Deep Domain-Adversarial Image Generation for Domain Generalisationcited by 231
  35. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  36. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  37. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  38. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 10; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 21
  2. 2025Out-of-Distribution Generalization in Time Series: A Surveycited by 16
  3. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalizationcited by 15
  4. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  5. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  6. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  7. 2024Unknown Domain Inconsistency Minimization for Domain Generalizationcited by 7
  8. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5
  9. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  10. 2025Error Slice Discovery via Manifold Compactnesscited by 4
  11. 2024SETA: Semantic-Aware Token Augmentation for Domain Generalizationcited by 3
  12. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 3
15 more
  1. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  2. 2024GAQAT: gradient-adaptive quantization-aware training for domain generalizationcited by 2
  3. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  4. 2024Meta Curvature-Aware Minimization for Domain Generalizationcited by 1
  5. 2025Adversarial Data Augmentation for Single Domain Generalization via Lyapunov Exponent-Guided Optimizationcited by 1
  6. 2025ODP-Bench: Benchmarking Out-of-Distribution Performance Predictioncited by 1
  7. 2026One-Step Generalization Ratio Guided Optimization for Domain Generalizationcited by 1
  8. 2023Seeking Flat Minima with Mean Teacher on Semi- and Weakly-Supervised Domain Generalization for Object Detectioncited by 0
  9. 2025Single Domain Generalization with Adversarial Memorycited by 0
  10. 2025DGSAM: Domain Generalization via Individual Sharpness-Aware Minimizationcited by 0
  11. 2025Beyond Deceptive Flatness: Dual-Order Solution for Strengthening Adversarial Transferabilitycited by 0
  12. 2025Generating Risky Samples with Conformity Constraints via Diffusion Modelscited by 0
  13. 2026Bi-Level Optimization for Single Domain Generalizationcited by 0
  14. 2026Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approachcited by 0
  15. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0