mlsanityData to 7 Oct 2026

BREEDS: Benchmarks for Subpopulation Shift

arXiv 2008.04859 ↗First posted 11 Aug 2020cs.CV cs.LG stat.MLcited by 155

Not on the map. Why?

Abstract (arXiv)

We develop a methodology for assessing the robustness of models to subpopulation shift---specifically, their ability to generalize to novel data subpopulations that were not observed during training. Our approach leverages the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. This enables us to synthesize realistic distribution shifts whose sources can be precisely controlled and characterized, within existing large-scale datasets. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity. We then validate that the corresponding shifts are tractable by obtaining human baselines for them. Finally, we utilize these benchmarks to measure the sensitivity of standard model architectures as well as the effectiveness of off-the-shelf train-time robustness interventions. Code and data available at https://github.com/MadryLab/BREEDS-Benchmarks .

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 40; 2023: 40; 2024: 33; 2025: 19; 2026: 8.

155 papers.

Sort:

By area:

  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  2. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  3. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  4. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  5. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  6. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  7. 2021Model-Based Domain Generalizationcited by 103
  8. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  9. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  10. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  11. 2022MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflictscited by 81
  12. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
38 more
  1. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  2. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 66
  3. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  4. 2021Mixture Outlier Exposure: Towards Out-of-Distribution Detection in Fine-grained Environmentscited by 65
  5. 2021Editing a classifier by rewriting its prediction rulescited by 64
  6. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 63
  7. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  8. 2024Beyond Model Adaptation at Test Time: A Surveycited by 55
  9. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  10. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  11. 2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
  12. 2023Fusing Models with Complementary Expertisecited by 45
  13. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  14. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  15. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
  16. 2023Improving neural network representations using human similarity judgmentscited by 38
  17. 2022Calibrated ensembles can mitigate accuracy tradeoffs under distribution shiftcited by 34
  18. 2022Invariant Feature Learning for Generalized Long-Tailed Classificationcited by 34
  19. 2022Domain Adaptation under Open Set Label Shiftcited by 32
  20. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  21. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  22. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  23. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  24. 2021Self-Supervised Classification Networkcited by 27
  25. 2021An Information-theoretic Approach to Distribution Shiftscited by 25
  26. 2023On the Connection between Pre-training Data Diversity and Fine-tuning Robustnesscited by 25
  27. 2024Any-Shift Prompting for Generalization over Distributionscited by 24
  28. 2022How to Fine-Tune Vision Models with SGDcited by 23
  29. 2021Clustering Mixture Models in Almost-Linear Time via List-Decodable Mean Estimationcited by 21
  30. 2023Adversarial Bayesian Augmentation for Single-Source Domain Generalizationcited by 21
  31. 2023HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understandingcited by 21
  32. 2023A Closer Look at Model Adaptation using Feature Distortion and Simplicity Biascited by 20
  33. 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
  34. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  35. 2023PAC Prediction Sets Under Label Shiftcited by 19
  36. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  37. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  38. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16

50 most cited of 155 All 155 (JSON)