# Estimating and Explaining Model Performance When Both Covariates and Labels Shift

arXiv: https://arxiv.org/abs/2209.08436 · HTML: https://mlsanity.com/paper/2209.08436/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 18 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Robust Fairness under Covariate Shift](https://mlsanity.com/paper/2010.05166/) (2020)
- [Estimating Generalization under Distribution Shifts via Domain-Invariant Representations](https://mlsanity.com/paper/2007.03511/) (2020)
- [Mandoline: Model Evaluation under Distribution Shift](https://mlsanity.com/paper/2107.00643/) (2021)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [Online Adaptation to Label Distribution Shift](https://mlsanity.com/paper/2107.04520/) (2021)
- [Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2003.00343/) (2020)
- [Active Learning under Label Shift](https://mlsanity.com/paper/2007.08479/) (2020)
- [Did the Model Change? Efficiently Assessing Machine Learning API Shifts](https://mlsanity.com/paper/2107.14203/) (2021)

## Built on by (most cited first, 19 of 19)

- [Energy-based Automated Model Evaluation](https://mlsanity.com/paper/2401.12689/) (2024)
- [Adapting to Continuous Covariate Shift via Online Density Ratio Estimation](https://mlsanity.com/paper/2302.02552/) (2023)
- [Explanation Shift: How Did the Distribution Shift Impact the Model?](https://mlsanity.com/paper/2303.08081/) (2023)
- [Binary Quantification and Dataset Shift: An Experimental Investigation](https://mlsanity.com/paper/2310.04565/) (2023)
- [Sparse joint shift in multinomial classification](https://mlsanity.com/paper/2303.16971/) (2023)
- [Prominent Roles of Conditionally Invariant Components in Domain Adaptation: Theory and Algorithms](https://mlsanity.com/paper/2309.10301/) (2023)
- [Nearest Neighbor Sampling for Covariate Shift Adaptation](https://mlsanity.com/paper/2312.09969/) (2023)
- [Invariance assumptions for class distribution estimation](https://mlsanity.com/paper/2311.17225/) (2023)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Fed-ADE: Adaptive Learning Rate for Federated Post-adaptation under Distribution Shift](https://mlsanity.com/paper/2603.01040/) (2026)
- [Active Multiple-Prediction-Powered Inference](https://mlsanity.com/paper/2605.08429/) (2026)
- [Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias](https://mlsanity.com/paper/2606.14506/) (2026)
- [Mining Invariance from Nonlinear Multi-Environment Data: Binary Classification](https://mlsanity.com/paper/2404.15245/) (2024)
- [A generalized approach to label shift: the Conditional Probability Shift Model](https://mlsanity.com/paper/2503.02583/) (2025)
- [Explaining Concept Shift with Interpretable Feature Attribution](https://mlsanity.com/paper/2505.20634/) (2025)
- [Factorizable joint shift revisited](https://mlsanity.com/paper/2601.15036/) (2026)
- [Causal Parametric Drift Simulation: A Digital Twin Framework for Classifier Robustness Evaluation](https://mlsanity.com/paper/2605.09663/) (2026)
- [Entropic Projection Alignment: Estimating, Explaining, and Improving Model Performance Under Distribution Shift](https://mlsanity.com/paper/2605.31250/) (2026)
- [Geometry-Aware Bayesian Quantification via Compositional Data Analysis](https://mlsanity.com/paper/2607.04977/) (2026)

## Abstract (arXiv)

Deployed machine learning (ML) models often encounter new user data that differs from their training data. Therefore, estimating how well a given model might perform on the new data is an important step toward reliable ML applications. This is very challenging, however, as the data distribution can change in flexible ways, and we may not have any labels on the new data, which is often the case in monitoring settings. In this paper, we propose a new distribution shift model, Sparse Joint Shift (SJS), which considers the joint shift of both labels and a few features. This unifies and generalizes several existing shift models including label shift and sparse covariate shift, where only marginal feature or label distribution shifts are considered. We describe mathematical conditions under which SJS is identifiable. We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate a model's performance on new data without any labels. We conduct extensive experiments on several real-world datasets with various ML models. Across different datasets and distribution shifts, SEES achieves significant (up to an order of magnitude) shift estimation error improvements over existing approaches.