mlsanityData to 14 Oct 2026

Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization

arXiv 2109.02934 ↗First posted 7 Sep 2021cs.LG cs.AI cs.CVcited by 186

Not on the map. Why?

Abstract (arXiv)

Learning robust models that generalize well under changes in the data distribution is critical for real-world applications. To this end, there has been a growing surge of interest to learn simultaneously from multiple training domains - while enforcing different types of invariance across those domains. Yet, all existing approaches fail to show systematic benefits under controlled evaluation protocols. In this paper, we introduce a new regularization - named Fishr - that enforces domain invariance in the space of the gradients of the loss: specifically, the domain-level variances of gradients are matched across training domains. Our approach is based on the close relations between the gradient covariance, the Fisher Information and the Hessian of the loss: in particular, we show that Fishr eventually aligns the domain-level loss landscapes locally around the final weights. Extensive experiments demonstrate the effectiveness of Fishr for out-of-distribution generalization. Notably, Fishr improves the state of the art on the DomainBed benchmark and performs consistently better than Empirical Risk Minimization. Our code is available at https://github.com/alexrame/fishr.

Key facts. Cited by 186 papers in the map (data through 14 Oct 2026). Builds on 93 of its 155 references that are in the map. First posted on arXiv 7 Sep 2021.

Builds on

93 of 155 references.

References · 93

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  9. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2019Invariant Risk Minimizationcited by 1,854
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
38 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  3. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
  4. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  6. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  7. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  8. 2020In Search of Lost Domain Generalizationcited by 1,054
  9. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  11. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
  12. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  13. 2018Recognition in Terra Incognitacited by 658
  14. 2013Domain Generalization via Invariant Feature Representationcited by 623
  15. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 600
  16. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
  17. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  18. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  19. 2014New insights and perspectives on the natural gradient methodcited by 443
  20. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  21. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  22. 2015Return of Frustratingly Easy Domain Adaptationcited by 391
  23. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  24. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  25. 2020Environment Inference for Invariant Learningcited by 306
  26. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  27. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  28. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  29. 2017Three Factors Influencing Minima in SGDcited by 273
  30. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
  31. 2020The Risks of Invariant Risk Minimizationcited by 262
  32. 2021Gradient Matching for Domain Generalizationcited by 258
  33. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  34. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  35. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  36. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  37. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  38. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 28; 2023: 56; 2024: 45; 2025: 31; 2026: 19.

186 papers.

Sort:

By area:

  1. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  2. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  3. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  4. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  5. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  6. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  7. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  8. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  9. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  10. 2023AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptationcited by 48
  11. 2023SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalizationcited by 48
  12. 2023Improved Test-Time Adaptation for Domain Generalizationcited by 47
38 more
  1. 2023Feature Alignment and Uniformity for Test Time Adaptationcited by 42
  2. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  3. 2022Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalizationcited by 38
  4. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  5. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  6. 2022Rich Feature Construction for the Optimization-Generalization Dilemmacited by 34
  7. 2023Free Lunch for Domain Adversarial Training: Environment Label Smoothingcited by 33
  8. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 33
  9. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  10. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  11. 2022Out-of-Distribution Representation Learning for Time Series Classificationcited by 31
  12. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 30
  13. 2022Domain Generalization by Learning and Removing Domain-specific Featurescited by 28
  14. 2023ALOFT: A Lightweight MLP-like Architecture with Dynamic Low-frequency Transform for Domain Generalizationcited by 28
  15. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  16. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  17. 2023Decompose, Adjust, Compose: Effective Normalization by Playing with Frequency for Domain Generalizationcited by 26
  18. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  19. 2023Fairness and Accuracy under Domain Generalizationcited by 24
  20. 2023DIVERSIFY: A General Framework for Time Series Out-of-distribution Detection and Generalizationcited by 24
  21. 2023Generalizable Decision Boundaries: Dualistic Meta-Learning for Open Set Domain Generalizationcited by 24
  22. 2023MADG: Margin-based Adversarial Learning for Domain Generalizationcited by 22
  23. 2023Towards Generalizable Diabetic Retinopathy Grading in Unseen Domainscited by 21
  24. 2023Spuriosity Didn't Kill the Classifier: Using Invariant Predictions to Harness Spurious Featurescited by 21
  25. 2023DomainDrop: Suppressing Domain-Sensitive Channels for Domain Generalizationcited by 21
  26. 2021MVDG: A Unified Multi-view Framework for Domain Generalizationcited by 20
  27. 2022Causal Balancing for Domain Generalizationcited by 20
  28. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  29. 2022Localized Adversarial Domain Generalizationcited by 19
  30. 2024Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlationcited by 18
  31. 2021Optimization-based Causal Estimation from Heterogenous Environmentscited by 16
  32. 2023Domain Generalization via Rationale Invariancecited by 16
  33. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  34. 2025FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentationcited by 16
  35. 2022Domain-Specific Risk Minimization for Out-of-Distribution Generalizationcited by 15
  36. 2023Understanding Hessian Alignment for Domain Generalizationcited by 15
  37. 2022Learning useful representations for shifting tasks and distributionscited by 14
  38. 2023Benchmarking Algorithms for Federated Domain Generalizationcited by 14

50 most cited of 186 All 186 (JSON)