# Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection

arXiv: https://arxiv.org/abs/2107.00191 · HTML: https://mlsanity.com/paper/2107.00191/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 1 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Improving model calibration with accuracy versus uncertainty optimization](https://mlsanity.com/paper/2012.07923/) (2020)
- [Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learning](https://mlsanity.com/paper/1904.11740/) (2019)
- [A linearized framework and a new benchmark for model selection for fine-tuning](https://mlsanity.com/paper/2102.00084/) (2021)
- [Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference](https://mlsanity.com/paper/1909.13550/) (2019)
- [Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics](https://mlsanity.com/paper/2101.10842/) (2021)
- [Layer-Specific Adaptive Learning Rates for Deep Networks](https://mlsanity.com/paper/1510.04609/) (2015)
- [Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization](https://mlsanity.com/paper/2001.06838/) (2020)
- [A Bayesian Perspective on Training Speed and Model Selection](https://mlsanity.com/paper/2010.14499/) (2020)
- [Detection of data drift and outliers affecting machine learning model performance over time](https://mlsanity.com/paper/2012.09258/) (2020)
- [Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation](https://mlsanity.com/paper/2002.11018/) (2020)

## Built on by (most cited first, 1 of 1)

- [DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Images](https://mlsanity.com/paper/2205.13723/) (2022)

## Abstract (arXiv)

While many real-world data streams imply that they change frequently in a nonstationary way, most of deep learning methods optimize neural networks on training data, and this leads to severe performance degradation when dataset shift happens. However, it is less possible to annotate or inspect newly streamed data by humans, and thus it is desired to measure model drift at inference time in an unsupervised manner. In this paper, we propose a novel method of model drift estimation by exploiting statistics of batch normalization layer on unlabeled test data. To remedy possible sampling error of streamed input data, we adopt low-rank approximation to each representational layer. We show the effectiveness of our method not only on dataset shift detection but also on model selection when there are multiple candidate models among model zoo or training trajectories in an unsupervised way. We further demonstrate the consistency of our method by comparing model drift scores between different network architectures.