mlsanityData to 7 Oct 2026

Assaying Out-Of-Distribution Generalization in Transfer Learning

arXiv 2207.09239 ↗First posted 19 Jul 2022cs.LG stat.MLcited by 53

Not on the map. Why?

Abstract (arXiv)

Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 102 of its 132 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

102 of 132 references.

References · 102

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2017On Calibration of Modern Neural Networkscited by 4,860
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2019Invariant Risk Minimizationcited by 1,832
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  13. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  14. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  15. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  16. 2018Model Cards for Model Reportingcited by 1,068
  17. 2020In Search of Lost Domain Generalizationcited by 1,042
  18. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  19. 2021Scaling Vision Transformerscited by 993
  20. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  21. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  22. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  23. 2020Object-Centric Learning with Slot Attentioncited by 865
  24. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  25. 2022A Generalist Agentcited by 802
  26. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  27. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  28. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  29. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  30. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  31. 2018Recognition in Terra Incognitacited by 655
  32. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  33. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  34. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  35. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  36. 2021Intriguing Properties of Vision Transformerscited by 446
  37. 2019Measuring Calibration in Deep Learningcited by 443
  38. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430

50 most cited of 102 All 102 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 23; 2024: 16; 2025: 5; 2026: 3.

53 papers.

Sort:

By area:

  1. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  2. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  3. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  4. 2023Exploring the Limits of Deep Image Clustering using Pretrained Modelscited by 33
  5. 2023Leveraging sparse and shared feature activations for disentangled representation learningcited by 32
  6. 2023HGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentationcited by 31
  7. 2024On the Challenges and Opportunities in Generative AIcited by 31
  8. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  9. 2022Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Taskscited by 29
  10. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  11. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  12. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
38 more
  1. 2023Robustifying Token Attention for Vision Transformerscited by 19
  2. 2022Neuromorphic Visual Scene Understanding with Resonator Networkscited by 17
  3. 2024Structure-based out-of-distribution (OOD) materials property prediction: a benchmark studycited by 17
  4. 2024Subgraph Pooling: Tackling Negative Transfer on Graphscited by 17
  5. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  6. 2023How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?cited by 14
  7. 2023Image retrieval outperforms diffusion models on data augmentationcited by 14
  8. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  9. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 13
  10. 2023Evaluating the Fairness of Discriminative Foundation Models in Computer Visioncited by 12
  11. 2023Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Linecited by 11
  12. 2024On the Generalization of Handwritten Text Recognition Modelscited by 10
  13. 2024Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisationcited by 9
  14. 2023Robust Representation Learning with Reliable Pseudo-labels Generation via Self-Adaptive Optimal Transport for Short Text Clusteringcited by 6
  15. 2023IBAFormer: Intra-batch Attention Transformer for Domain Generalized Semantic Segmentationcited by 6
  16. 2024What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?cited by 6
  17. 2023Meaningful Causal Aggregation and Paradoxical Confoundingcited by 5
  18. 2024BehAVE: Behaviour Alignment of Video Game Encodingscited by 5
  19. 2024Scaling Laws for Galaxy Imagescited by 5
  20. 2023Out-of-Variable Generalization for Discriminative Modelscited by 4
  21. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  22. 2023Unraveling the Key Components of OOD Generalization via Diversificationcited by 4
  23. 2026Out-of-Distribution Generalization for Neural Physics Solverscited by 4
  24. 2022Empirical Study on Optimizer Selection for Out-of-Distribution Generalizationcited by 3
  25. 2023Finding Competence Regions in Domain Generalizationcited by 3
  26. 2023Robustness May be More Brittle than We Think under Different Degrees of Distribution Shiftscited by 2
  27. 2024In-context Learning in Presence of Spurious Correlationscited by 2
  28. 2024Causality-Driven Audits of Model Robustnesscited by 2
  29. 2025Supervised Contrastive Block Disentanglementcited by 2
  30. 2025Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlationscited by 2
  31. 2023Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetriescited by 1
  32. 2023Benchmarking Low-Shot Robustness to Natural Distribution Shiftscited by 1
  33. 2024Uncertainty modeling for fine-tuned implicit functionscited by 1
  34. 2024Introducing 'Inside' Out of Distributioncited by 1
  35. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  36. 2025RockNet: Distributed Learning on Ultra-Low-Power Devicescited by 1
  37. 2026Measuring Distribution Shift in User Prompts and Its Effects on LLM Performancecited by 1
  38. 2023Adapting Contrastive Language-Image Pretrained (CLIP) Models for Out-of-Distribution Detectioncited by 0

50 most cited of 53 All 53 (JSON)