# Backward-Compatible Prediction Updates: A Probabilistic Approach

arXiv: https://arxiv.org/abs/2107.01057 · HTML: https://mlsanity.com/paper/2107.01057/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [Kernel-based Tests for Joint Independence](https://mlsanity.com/paper/1603.00285/) (2016)
- [An Empirical Analysis of Backward Compatibility in Machine Learning Systems](https://mlsanity.com/paper/2008.04572/) (2020)
- [Fixing the train-test resolution discrepancy: FixEfficientNet](https://mlsanity.com/paper/2003.08237/) (2020)
- [Active Learning from Imperfect Labelers](https://mlsanity.com/paper/1610.09730/) (2016)

## Built on by (most cited first, 14 of 14)

- [Eureka: Evaluating and Understanding Large Foundation Models](https://mlsanity.com/paper/2409.10566/) (2024)
- [Predictive Churn with the Set of Good Models](https://mlsanity.com/paper/2402.07745/) (2024)
- [MUSCLE: A Model Update Strategy for Compatible LLM Evolution](https://mlsanity.com/paper/2407.09435/) (2024)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environment](https://mlsanity.com/paper/2406.13371/) (2024)
- [Measuring and Reducing Model Update Regression in Structured Prediction for NLP](https://mlsanity.com/paper/2202.02976/) (2022)
- [BT2: Backward-compatible Training with Basis Transformation](https://mlsanity.com/paper/2211.03989/) (2022)
- [ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training](https://mlsanity.com/paper/2205.06265/) (2022)
- [FastFill: Efficient Compatible Model Update](https://mlsanity.com/paper/2303.04766/) (2023)
- [Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion](https://mlsanity.com/paper/2302.02080/) (2023)
- [Robustness-Congruent Adversarial Training for Secure Machine Learning Model Updates](https://mlsanity.com/paper/2402.17390/) (2024)
- [Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA](https://mlsanity.com/paper/2608.22856/) (2026)
- [Reference-Tail Trust:Certified Probability Floors for Learned Updates Inside a Deployed Network](https://mlsanity.com/paper/2609.34904/) (2026)
- [SAGE: A Statistical Acceptance Gate for Self-Evolving Agents](https://mlsanity.com/paper/2609.36043/) (2026)

## Abstract (arXiv)

When machine learning systems meet real world applications, accuracy is only one of several requirements. In this paper, we assay a complementary perspective originating from the increasing availability of pre-trained and regularly improving state-of-the-art models. While new improved models develop at a fast pace, downstream tasks vary more slowly or stay constant. Assume that we have a large unlabelled data set for which we want to maintain accurate predictions. Whenever a new and presumably better ML models becomes available, we encounter two problems: (i) given a limited budget, which data points should be re-evaluated using the new model?; and (ii) if the new predictions differ from the current ones, should we update? Problem (i) is about compute cost, which matters for very large data sets and models. Problem (ii) is about maintaining consistency of the predictions, which can be highly relevant for downstream applications; our demand is to avoid negative flips, i.e., changing correct to incorrect predictions. In this paper, we formalize the Prediction Update Problem and present an efficient probabilistic approach as answer to the above questions. In extensive experiments on standard classification benchmark data sets, we show that our method outperforms alternative strategies along key metrics for backward-compatible prediction updates.