# Contrastive Identification of Covariate Shift in Image Data

arXiv: https://arxiv.org/abs/2108.08000 · HTML: https://mlsanity.com/paper/2108.08000/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning](https://mlsanity.com/paper/1908.00087/) (2019)
- [OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples](https://mlsanity.com/paper/2002.03103/) (2020)
- [Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration](https://mlsanity.com/paper/1610.09064/) (2016)
- [Diagnosing Concept Drift with Visual Analytics](https://mlsanity.com/paper/2007.14372/) (2020)
- [ConceptExplorer: Visual Analysis of Concept Driftsin Multi-source Time-series Data](https://mlsanity.com/paper/2007.15272/) (2020)

## Built on by (most cited first, 4 of 4)

- [DendroMap: Visual Exploration of Large-Scale Image Datasets for Machine Learning with Treemaps](https://mlsanity.com/paper/2205.06935/) (2022)
- [DeepLens: Interactive Out-of-distribution Data Detection in NLP Models](https://mlsanity.com/paper/2303.01577/) (2023)
- [ESCAPE: Countering Systematic Errors from Machine's Blind Spots via Interactive Visual Analysis](https://mlsanity.com/paper/2303.09657/) (2023)
- [Cross-GAN Auditing: Unsupervised Identification of Attribute Level Similarities and Differences between Pretrained Generative Models](https://mlsanity.com/paper/2303.10774/) (2023)

## Abstract (arXiv)

Identifying covariate shift is crucial for making machine learning systems robust in the real world and for detecting training data biases that are not reflected in test data. However, detecting covariate shift is challenging, especially when the data consists of high-dimensional images, and when multiple types of localized covariate shift affect different subspaces of the data. Although automated techniques can be used to detect the existence of covariate shift, our goal is to help human users characterize the extent of covariate shift in large image datasets with interfaces that seamlessly integrate information obtained from the detection algorithms. In this paper, we design and evaluate a new visual interface that facilitates the comparison of the local distributions of training and test data. We conduct a quantitative user study on multi-attribute facial data to compare two different learned low-dimensional latent representations (pretrained ImageNet CNN vs. density ratio) and two user analytic workflows (nearest-neighbor vs. cluster-to-cluster). Our results indicate that the latent representation of our density ratio model, combined with a nearest-neighbor comparison, is the most effective at helping humans identify covariate shift.