mlsanityData to 7 Oct 2026

Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

arXiv 2107.04649 ↗First posted 9 Jul 2021cs.LG stat.MLcited by 248

Not on the map. Why?

Abstract (arXiv)

For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Builds on 82 of its 123 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

82 of 123 references.

References · 82

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2016Pyramid Scene Parsing Networkcited by 2,472
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  12. 2018Deep contextualized word representationscited by 2,285
  13. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  14. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  15. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  16. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
  17. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  18. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  19. 2019Natural Adversarial Examplescited by 1,401
  20. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  21. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  22. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  23. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  24. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  25. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  26. 2020In Search of Lost Domain Generalizationcited by 1,042
  27. 2020Designing Network Design Spacescited by 954
  28. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  29. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  30. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  31. 2018Do Better ImageNet Models Transfer Better?cited by 560
  32. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  33. 2017Progressive Neural Architecture Searchcited by 483
  34. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  35. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  36. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  37. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  38. 2017Functional Map of the Worldcited by 338

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 59; 2023: 68; 2024: 58; 2025: 33; 2026: 21.

248 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2021Robust fine-tuning of zero-shot modelscited by 785
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  6. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  7. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  8. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  9. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  10. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  11. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  12. 20223D Common Corruptions and Data Augmentationcited by 120
38 more
  1. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  2. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  3. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  4. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  5. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  6. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  7. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  8. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  9. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  10. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  11. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  12. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  13. 2022Human alignment of neural network representationscited by 74
  14. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  15. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  16. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  17. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  18. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  19. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  20. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  21. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  22. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  23. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  24. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  25. 2023Describing Differences in Image Sets with Natural Languagecited by 51
  26. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  27. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  28. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  29. 2024Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Drivingcited by 39
  30. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  31. 2023Diagnosing Model Performance Under Distribution Shiftcited by 37
  32. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  33. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  34. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 35
  35. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  36. 2024Proactive Conversational Agents with Inner Thoughtscited by 35
  37. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34
  38. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33

50 most cited of 248 All 248 (JSON)