# A unified framework for dataset shift diagnostics

arXiv: https://arxiv.org/abs/2205.08340 · HTML: https://mlsanity.com/paper/2205.08340/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 77 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 26 of 26)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Characterizing Concept Drift](https://mlsanity.com/paper/1511.03816/) (2015)
- [Approximating Likelihood Ratios with Calibrated Discriminative Classifiers](https://mlsanity.com/paper/1506.02169/) (2015)
- [Amortised MAP Inference for Image Super-resolution](https://mlsanity.com/paper/1610.04490/) (2016)
- [Diagnosing failures of fairness transfer across distribution shift in real-world medical settings](https://mlsanity.com/paper/2202.01034/) (2022)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Conditional Independence Testing using Generative Adversarial Networks](https://mlsanity.com/paper/1907.04068/) (2019)
- [Converting High-Dimensional Regression to High-Dimensional Conditional Density Estimation](https://mlsanity.com/paper/1704.08095/) (2017)
- [Local permutation tests for conditional independence](https://mlsanity.com/paper/2112.11666/) (2021)
- [DenseNet Models for Tiny ImageNet Classification](https://mlsanity.com/paper/1904.10429/) (2019)
- [A linear adjustment based approach to posterior drift in transfer learning](https://mlsanity.com/paper/2111.10841/) (2021)
- [Quantification under prior probability shift: the ratio estimator and its extensions](https://mlsanity.com/paper/1807.03929/) (2018)
- [Testing for concept shift online](https://mlsanity.com/paper/2012.14246/) (2020)
- [Effective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptation](https://mlsanity.com/paper/2010.01184/) (2020)
- [Failing Conceptually: Concept-Based Explanations of Dataset Shift](https://mlsanity.com/paper/2104.08952/) (2021)
- [Concept Drift Detection and Adaptation with Hierarchical Hypothesis Testing](https://mlsanity.com/paper/1707.07821/) (2017)
- [Concept Drift and Covariate Shift Detection Ensemble with Lagged Labels](https://mlsanity.com/paper/2012.04759/) (2020)
- [Dataset shift quantification for credit card fraud detection](https://mlsanity.com/paper/1906.06977/) (2019)
- [Predicting Legal Proceedings Status: Approaches Based on Sequential Text Data](https://mlsanity.com/paper/2003.11561/) (2020)
- [OCDE: Odds Conditional Density Estimator](https://mlsanity.com/paper/2107.04118/) (2021)

## Built on by (most cited first, 7 of 7)

- [Information Suppression in Large Language Models: Auditing, Quantifying, and Characterizing Censorship in DeepSeek](https://mlsanity.com/paper/2506.12349/) (2025)
- [Conditional independence testing under misspecified inductive biases](https://mlsanity.com/paper/2307.02520/) (2023)
- [Efficient and Multiply Robust Risk Estimation under General Forms of Dataset Shift](https://mlsanity.com/paper/2306.16406/) (2023)
- [DriftGuard: Safety-Aware Multi-Monitor Detection and Selective Adaptation for Evolving Toxicity Moderation](https://mlsanity.com/paper/2606.28725/) (2026)
- [Quantifying Data Similarity Using Cross Learning](https://mlsanity.com/paper/2510.10866/) (2025)
- [Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference](https://mlsanity.com/paper/2402.05330/) (2024)
- [Computing Marginal and Conditional Divergences between Decomposable Models with Applications](https://mlsanity.com/paper/2310.09129/) (2023)

## Abstract (arXiv)

Supervised learning techniques typically assume training data originates from the target population. Yet, in reality, dataset shift frequently arises, which, if not adequately taken into account, may decrease the performance of their predictors. In this work, we propose a novel and flexible framework called DetectShift that quantifies and tests for multiple dataset shifts, encompassing shifts in the distributions of (X, Y), X, Y, X|Y, and Y|X. DetectShift equips practitioners with insights into data shifts, facilitating the adaptation or retraining of predictors using both source and target data. This proves extremely valuable when labeled samples in the target domain are limited. The framework utilizes test statistics with the same nature to quantify the magnitude of the various shifts, making results more interpretable. It is versatile, suitable for regression and classification tasks, and accommodates diverse data forms - tabular, text, or image. Experimental results demonstrate the effectiveness of DetectShift in detecting dataset shifts even in higher dimensions.