mlsanityData to 14 Oct 2026

WILDS: A Benchmark of in-the-Wild Distribution Shifts

arXiv 2012.07421 ↗First posted 14 Dec 2020cs.LGcited by 1,375

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.

Key facts. Cited by 1,375 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 120; 2022: 274; 2023: 279; 2024: 250; 2025: 222; 2026: 230.

1,375 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  3. 2021Robust fine-tuning of zero-shot modelscited by 797
  4. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  5. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  6. 2021Domain Generalization: A Surveycited by 764
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  9. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  10. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  11. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
38 more
  1. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  2. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 415
  3. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  5. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  6. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377
  7. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  8. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  9. 2021Unsolved Problems in ML Safetycited by 292
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  11. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  12. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  13. 2022Prompting GPT-3 To Be Reliablecited by 260
  14. 2021Gradient Matching for Domain Generalizationcited by 258
  15. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  16. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  17. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  18. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  19. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  20. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  21. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  22. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  23. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  24. 2021Simple data balancing achieves competitive worst-group-accuracycited by 178
  25. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  26. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  27. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  28. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  29. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  30. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  31. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  32. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
  33. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  34. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  35. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  36. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  37. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  38. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145

50 most cited of 1,375 All 1,375 (JSON)