# Mitigating covariate shift in non-colocated data with learned parameter priors

arXiv: https://arxiv.org/abs/2411.06499 · HTML: https://mlsanity.com/paper/2411.06499/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [Rethinking Importance Weighting for Deep Learning under Distribution Shift](https://mlsanity.com/paper/2006.04662/) (2020)
- [Cold Case: The Lost MNIST Digits](https://mlsanity.com/paper/1905.10498/) (2019)
- [A One-step Approach to Covariate Shift Adaptation](https://mlsanity.com/paper/2007.04043/) (2020)
- [Adapting to Continuous Covariate Shift via Online Density Ratio Estimation](https://mlsanity.com/paper/2302.02552/) (2023)
- [Kannada-MNIST: A new handwritten digits dataset for the Kannada language](https://mlsanity.com/paper/1908.01242/) (2019)
- [A New Lower Bound for Kullback-Leibler Divergence Based on Hammersley-Chapman-Robbins Bound](https://mlsanity.com/paper/1907.00288/) (2019)
- [Insights into Fairness through Trust: Multi-scale Trust Quantification for Financial Deep Learning](https://mlsanity.com/paper/2011.01961/) (2020)

## Built on by (most cited first, 1 of 1)

- [CalTwin: Towards Calibrated, Shift-Robust Medical World Models via Fisher-Information Regularisation](https://mlsanity.com/paper/2607.26752/) (2026)

## Abstract (arXiv)

When training data are distributed across time or space, covariate shift across fragments of training data biases cross-validation, compromising model selection and assessment. We present textitFragmentation-Induced covariate-shift Remediation (FIcsR), which minimizes an f-divergence between a fragment's covariate distribution and that of the standard cross-validation baseline. We show an equivalence with popular importance-weighting methods. The method's numerical solution poses a computational challenge owing to the overparametrized nature of a neural network, and we derive a Fisher Information approximation. When accumulated over fragments, this provides a global estimate of the amount of shift remediation thus far needed, and we incorporate that as a prior via the minimization objective. In the paper, we run extensive classification experiments on multiple data classes, over 40 datasets, and with data batched over multiple sequence lengths. We extend the study to the k-fold cross-validation setting through a similar set of experiments. An ablation study exposes the method to varying amounts of shift and demonstrates slower degradation with FIcsR in place. The results are promising under all these conditions; with improved accuracy against batch and fold state-of-the-art by more than 5% and 10%, respectively.