# Shifts 2.0: Extending The Dataset of Real Distributional Shifts

arXiv: https://arxiv.org/abs/2206.15407 · HTML: https://mlsanity.com/paper/2206.15407/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 80 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Builds on (most cited first, 30 of 31)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?](https://mlsanity.com/paper/2006.14911/) (2020)
- [Automated brain extraction of multi-sequence MRI using artificial neural networks](https://mlsanity.com/paper/1901.11341/) (2019)
- [Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches](https://mlsanity.com/paper/1803.04386/) (2018)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Extending the WILDS Benchmark for Unsupervised Adaptation](https://mlsanity.com/paper/2112.05090/) (2021)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks](https://mlsanity.com/paper/1912.10481/) (2019)
- [Common Limitations of Image Processing Metrics: A Picture Story](https://mlsanity.com/paper/2104.05642/) (2021)
- [MTNT: A Testbed for Machine Translation of Noisy Text](https://mlsanity.com/paper/1809.00388/) (2018)
- [Neural-Symbolic Integration: A Compositional Perspective](https://mlsanity.com/paper/2010.11926/) (2020)

## Built on by (most cited first, 24 of 24)

- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift](https://mlsanity.com/paper/2406.00661/) (2024)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [Novel structural-scale uncertainty measures and error retention curves: application to multiple sclerosis](https://mlsanity.com/paper/2211.04825/) (2022)
- [Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentation](https://mlsanity.com/paper/2302.05432/) (2023)
- [HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts](https://mlsanity.com/paper/2305.18421/) (2023)
- [Evaluating Robustness and Uncertainty of Graph Models Under Structural Distributional Shifts](https://mlsanity.com/paper/2302.13875/) (2023)
- [Transfer Learning Study of Motion Transformer-based Trajectory Predictions](https://mlsanity.com/paper/2404.08271/) (2024)
- [Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation](https://mlsanity.com/paper/2308.07506/) (2023)
- [A Universal Metric of Dataset Similarity for Cross-silo Federated Learning](https://mlsanity.com/paper/2404.18773/) (2024)
- [FAIR Universe HiggsML Uncertainty Dataset and Competition](https://mlsanity.com/paper/2410.02867/) (2024)
- [UNISELF: A Unified Network with Instance Normalization and Self-Ensembled Lesion Fusion for Multiple Sclerosis Lesion Segmentation](https://mlsanity.com/paper/2508.03982/) (2025)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Diversifying Deep Ensembles: A Saliency Map Approach for Enhanced OOD Detection, Calibration, and Accuracy](https://mlsanity.com/paper/2305.11616/) (2023)
- [Fast refacing of MR images with a generative neural network lowers re-identification risk and preserves volumetric consistency](https://mlsanity.com/paper/2305.16922/) (2023)
- [Predictive uncertainty estimation in deep learning for lung carcinoma classification in digital pathology under real dataset shifts](https://mlsanity.com/paper/2408.08432/) (2024)
- [Towards Better Generalization via Distributional Input Projection Network](https://mlsanity.com/paper/2506.04690/) (2025)
- [Self-supervised Brain Lesion Generation for Effective Data Augmentation of Medical Images](https://mlsanity.com/paper/2406.14826/) (2024)
- [Interpretability of Uncertainty: Exploring Cortical Lesion Segmentation in Multiple Sclerosis](https://mlsanity.com/paper/2407.05761/) (2024)
- [Clinnova Federated Learning Proof of Concept: Key Takeaways from a Cross-border Collaboration](https://mlsanity.com/paper/2410.02443/) (2024)
- [Provable Robust Overfitting Mitigation in Wasserstein Distributionally Robust Optimization](https://mlsanity.com/paper/2503.04315/) (2025)
- [F3-Net: Foundation Model for Full Abnormality Segmentation of Medical Images with Flexible Input Modality Requirement](https://mlsanity.com/paper/2507.08460/) (2025)

## Abstract (arXiv)

Distributional shift, or the mismatch between training and deployment data, is a significant obstacle to the usage of machine learning in high-stakes industrial applications, such as autonomous driving and medicine. This creates a need to be able to assess how robustly ML models generalize as well as the quality of their uncertainty estimates. Standard ML baseline datasets do not allow these properties to be assessed, as the training, validation and test data are often identically distributed. Recently, a range of dedicated benchmarks have appeared, featuring both distributionally matched and shifted data. Among these benchmarks, the Shifts dataset stands out in terms of the diversity of tasks as well as the data modalities it features. While most of the benchmarks are heavily dominated by 2D image classification tasks, Shifts contains tabular weather forecasting, machine translation, and vehicle motion prediction tasks. This enables the robustness properties of models to be assessed on a diverse set of industrial-scale tasks and either universal or directly applicable task-specific conclusions to be reached. In this paper, we extend the Shifts Dataset with two datasets sourced from industrial, high-risk applications of high societal importance. Specifically, we consider the tasks of segmentation of white matter Multiple Sclerosis lesions in 3D magnetic resonance brain images and the estimation of power consumption in marine cargo vessels. Both tasks feature ubiquitous distributional shifts and a strict safety requirement due to the high cost of errors. These new datasets will allow researchers to further explore robust generalization and uncertainty estimation in new situations. In this work, we provide a description of the dataset and baseline results for both tasks.