# A Unified View of Label Shift Estimation

arXiv: https://arxiv.org/abs/2003.07554 · HTML: https://mlsanity.com/paper/2003.07554/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2020.

## Built on by (most cited first, 30 of 119)

- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Mixture Proportion Estimation and PU Learning: A Modern Approach](https://mlsanity.com/paper/2111.00980/) (2021)
- [Diagnosing failures of fairness transfer across distribution shift in real-world medical settings](https://mlsanity.com/paper/2202.01034/) (2022)
- [Online Adaptation to Label Distribution Shift](https://mlsanity.com/paper/2107.04520/) (2021)
- [Subspace Identification for Multi-Source Domain Adaptation](https://mlsanity.com/paper/2310.04723/) (2023)
- [Domain Adaptation under Open Set Label Shift](https://mlsanity.com/paper/2207.13048/) (2022)
- [RLSbench: Domain Adaptation Under Relaxed Label Shift](https://mlsanity.com/paper/2302.03020/) (2023)
- [Conformal Prediction for Federated Uncertainty Quantification Under Label Shift](https://mlsanity.com/paper/2306.05131/) (2023)
- [Adapting to Online Label Shift with Provable Guarantees](https://mlsanity.com/paper/2207.02121/) (2022)
- [Aggregating From Multiple Target-Shifted Sources](https://mlsanity.com/paper/2105.04051/) (2021)
- [Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation](https://mlsanity.com/paper/2302.01094/) (2023)
- [PAC Prediction Sets Under Label Shift](https://mlsanity.com/paper/2310.12964/) (2023)
- [A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization](https://mlsanity.com/paper/2111.02355/) (2021)
- [Deconstructing Distributions: A Pointwise Framework of Learning](https://mlsanity.com/paper/2202.09931/) (2022)
- [Mapping conditional distributions for domain adaptation under generalized target shift](https://mlsanity.com/paper/2110.15057/) (2021)
- [GeoNet: Benchmarking Unsupervised Adaptation across Geographies](https://mlsanity.com/paper/2303.15443/) (2023)
- [Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models](https://mlsanity.com/paper/2111.13657/) (2021)
- [Adapting to Continuous Covariate Shift via Online Density Ratio Estimation](https://mlsanity.com/paper/2302.02552/) (2023)
- [MLDemon: Deployment Monitoring for Machine Learning Systems](https://mlsanity.com/paper/2104.13621/) (2021)
- [Contrastive Attraction and Contrastive Repulsion for Representation Learning](https://mlsanity.com/paper/2105.03746/) (2021)
- [Domain Adaptation under Missingness Shift](https://mlsanity.com/paper/2211.02093/) (2022)
- [Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2211.10412/) (2022)
- [Adapting to Latent Subgroup Shifts via Concepts and Proxies](https://mlsanity.com/paper/2212.11254/) (2022)
- [Understanding new tasks through the lens of training data via exponential tilting](https://mlsanity.com/paper/2205.13577/) (2022)
- [Online Label Shift: Optimal Dynamic Regret meets Practical Algorithms](https://mlsanity.com/paper/2305.19570/) (2023)
- [Adaptive Training Distributions with Scalable Online Bilevel Optimization](https://mlsanity.com/paper/2311.11973/) (2023)

## Abstract (arXiv)

Under label shift, the label distribution p(y) might change but the class-conditional distributions p(x|y) do not. There are two dominant approaches for estimating the label marginal. BBSE, a moment-matching approach based on confusion matrices, is provably consistent and provides interpretable error bounds. However, a maximum likelihood estimation approach, which we call MLLS, dominates empirically. In this paper, we present a unified view of the two methods and the first theoretical characterization of MLLS. Our contributions include (i) consistency conditions for MLLS, which include calibration of the classifier and a confusion matrix invertibility condition that BBSE also requires; (ii) a unified framework, casting BBSE as roughly equivalent to MLLS for a particular choice of calibration method; and (iii) a decomposition of MLLS's finite-sample error into terms reflecting miscalibration and estimation error. Our analysis attributes BBSE's statistical inefficiency to a loss of information due to coarse calibration. Experiments on synthetic data, MNIST, and CIFAR10 support our findings.