mlsanityData to 7 Oct 2026

A Fine-Grained Analysis on Distribution Shift

arXiv 2110.11328 ↗First posted 21 Oct 2021cs.LG cs.CVcited by 171

Not on the map. Why?

Abstract (arXiv)

Robustness to distribution shifts is critical for deploying machine learning models in the real world. Despite this necessity, there has been little work in defining the underlying mechanisms that cause these shifts and evaluating the robustness of algorithms across multiple, different distribution shifts. To this end, we introduce a framework that enables fine-grained analysis of various distribution shifts. We provide a holistic analysis of current state-of-the-art methods by evaluating 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Overall, we train more than 85K models. Our experimental framework can be easily extended to include new methods, shifts, and datasets. We find, unlike previous work~citepGulrajani20, that progress has been made over a standard ERM baseline; in particular, pretraining and augmentations (learned or heuristic) offer large gains in many cases. However, the best methods are not consistent over different datasets and shifts.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Builds on 61 of its 93 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

61 of 93 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
38 more
  1. 2019Invariant Risk Minimizationcited by 1,832
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  6. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  8. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  9. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  10. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  11. 2020In Search of Lost Domain Generalizationcited by 1,042
  12. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  13. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  14. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  15. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  16. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  17. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  18. 2018Recognition in Terra Incognitacited by 655
  19. 2018Disentangling by Factorisingcited by 614
  20. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 609
  21. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  22. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  23. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  24. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  25. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  26. 2018Understanding disentangling in β-VAEcited by 332
  27. 2019Adversarial Domain Adaptation with Domain Mixupcited by 250
  28. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  29. 2020Deep Domain-Adversarial Image Generation for Domain Generalisationcited by 231
  30. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  31. 2019Causality matters in medical imagingcited by 193
  32. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  33. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  34. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  35. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  36. 2018Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttimecited by 122
  37. 2020Heterogeneous Domain Generalization via Domain Mixupcited by 103
  38. 2019On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Datasetcited by 100

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 47; 2023: 59; 2024: 31; 2025: 24; 2026: 9.

171 papers.

Sort:

By area:

  1. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  2. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  3. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  4. 2024Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factorscited by 106
  5. 2023Generative models improve fairness of medical classifiers under distribution shiftscited by 90
  6. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  7. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  8. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  9. 2023On Pitfalls of Test-Time Adaptationcited by 63
  10. 2023No Representation Rules Them All in Category Discoverycited by 61
  11. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  12. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
38 more
  1. 2022RankFeat: Rank-1 Feature Removal for Out-of-distribution Detectioncited by 52
  2. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  3. 2022Mechanistic Mode Connectivitycited by 50
  4. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  5. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  6. 2022Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shiftcited by 38
  7. 2022ActMAD: Activation Matching to Align Distributions for Test-Time-Trainingcited by 38
  8. 2022Learning Domain Invariant Prompt for Vision-Language Modelscited by 35
  9. 2022Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalizationcited by 34
  10. 2022Reliable Representations Make A Stronger Defender: Unsupervised Structure Refinement for Robust GNNcited by 34
  11. 2022Provable Domain Generalization via Invariant-Feature Subspace Recoverycited by 32
  12. 2023Leveraging sparse and shared feature activations for disentangled representation learningcited by 32
  13. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  14. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  15. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  16. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  17. 2022Operationalizing Machine Learning: An Interview Studycited by 25
  18. 2024Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamicscited by 25
  19. 2023Out-of-Domain Robustness via Targeted Augmentationscited by 24
  20. 2023ManyDG: Many-domain Generalization for Healthcare Applicationscited by 23
  21. 2024Adapting to Distribution Shift by Visual Domain Prompt Generationcited by 23
  22. 2022Transferring Fairness under Distribution Shifts via Fair Consistency Regularizationcited by 21
  23. 2023Improving Lens Flare Removal with General Purpose Pipeline and Multiple Light Sources Recoverycited by 21
  24. 2022Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalizationcited by 20
  25. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  26. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  27. 2023Diversifying Spatial-Temporal Perception for Video Domain Generalizationcited by 19
  28. 2022When Does Group Invariant Learning Survive Spurious Correlations?cited by 18
  29. 2022"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shiftscited by 18
  30. 2023Gradual Domain Adaptation: Theory and Algorithmscited by 18
  31. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  32. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  33. 2022Evaluating Latent Space Robustness and Uncertainty of EEG-ML Models under Realistic Distribution Shiftscited by 16
  34. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  35. 2024The Data Addition Dilemmacited by 15
  36. 2023How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?cited by 14
  37. 2023Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalizationcited by 14
  38. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learningcited by 14

50 most cited of 171 All 171 (JSON)