mlsanityData to 14 Oct 2026

Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters

arXiv 2310.07361 ↗First posted 11 Oct 2023cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Overfitting to the source domain is a common issue in gradient-based training of deep neural networks. To compensate for the over-parameterized models, numerous regularization techniques have been introduced such as those based on dropout. While these methods achieve significant improvements on classical benchmarks such as ImageNet, their performance diminishes with the introduction of domain shift in the test set i.e. when the unseen data comes from a significantly different distribution. In this paper, we move away from the classical approach of Bernoulli sampled dropout mask construction and propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network's parameters. Specifically, at each training step, parameters with high GSNR will be discarded. Furthermore, we alleviate the burden of manually searching for the optimal dropout ratio by leveraging a meta-learning approach. We evaluate our method on standard domain generalization benchmarks and achieve competitive results on classification and face anti-spoofing problems.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 59 of its 89 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

59 of 89 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  6. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  9. 2018On the Spectral Bias of Neural Networkscited by 1,341
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  11. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  12. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
38 more
  1. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 683
  2. 2013Domain Generalization via Invariant Feature Representationcited by 623
  3. 2017Self-Normalizing Neural Networkscited by 598
  4. 2021Domain Generalization with MixStylecited by 566
  5. 2019Domain Generalization by Solving Jigsaw Puzzlescited by 450
  6. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  7. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 407
  8. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 379
  9. 2019Semi-supervised Domain Adaptation via Minimax Entropycited by 335
  10. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 323
  11. 2020Learning to Generate Novel Domains for Domain Generalizationcited by 296
  12. 2018Generalizing Across Domains via Cross-Gradient Trainingcited by 268
  13. 2018DropBlock: A regularization method for convolutional networkscited by 262
  14. 2014Efficient Object Localization Using Convolutional Networkscited by 227
  15. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  16. 2019Domain-Specific Batch Normalization for Unsupervised Domain Adaptationcited by 197
  17. 2019Domain Generalization Using a Mixture of Multiple Latent Domainscited by 170
  18. 2021Learning to Diversify for Single Domain Generalizationcited by 170
  19. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 161
  20. 2020Domain Adaptive Ensemble Learningcited by 156
  21. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  22. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  23. 2020Robust and Generalizable Visual Representation Learning via Random Convolutionscited by 142
  24. 2020Single-Side Domain Generalization for Face Anti-Spoofingcited by 139
  25. 2020Learning to Balance Specificity and Invariance for In and Out of Domain Generalizationcited by 128
  26. 2021Open Domain Generalization with Domain-Augmented Meta-Learningcited by 115
  27. 2018Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shiftcited by 109
  28. 2020Cross-domain Face Presentation Attack Detection via Multi-domain Disentangled Representation Learningcited by 93
  29. 2019Regularized Fine-grained Meta Face Anti-spoofingcited by 86
  30. 2022Domain Generalization via Shuffled Style Assembly for Face Anti-Spoofingcited by 73
  31. 2022A Broad Study of Pre-training for Domain Generalization and Adaptationcited by 65
  32. 2021Self-Domain Adaptation for Face Anti-Spoofingcited by 49
  33. 2021Adaptive Normalized Representation Learning for Generalizable Face Anti-Spoofingcited by 49
  34. 2021Dual Reweighting Domain Generalization for Face Presentation Attack Detectioncited by 47
  35. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  36. 2021Generalizable Representation Learning for Mixture Domain Face Anti-Spoofingcited by 46
  37. 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
  38. 2018Differentiable Learning-to-Normalize via Switchable Normalizationcited by 33

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2024A Layer Selection Approach to Test Time Adaptationcited by 7
  2. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainercited by 4
  3. 2025Energy-Aware Deep Learning on Resource-Constrained Hardwarecited by 3
  4. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  5. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  6. 2025Graph Coloring for Multi-Task Learningcited by 1
  7. 2026One-Step Generalization Ratio Guided Optimization for Domain Generalizationcited by 1
  8. 2025Generative Classifier for Domain Generalizationcited by 0
  9. 2026Federated Learning for Feature Generalization with Convex Constraintscited by 0