# Backward Compatibility During Data Updates by Weight Interpolation arXiv: https://arxiv.org/abs/2301.10546 ยท HTML: https://mlsanity.com/paper/2301.10546/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 46 references that are in the map. - First posted on arXiv 25 Jan 2023. ## Builds on (most cited first, 30 of 34) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016) - [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016) - [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015) - [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017) - [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022) - [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018) - [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021) - [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018) - [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022) - [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021) - [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018) - [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021) - [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021) - [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021) - [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022) - [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020) - [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020) - [Patching open-vocabulary models by interpolating weights](https://mlsanity.com/paper/2208.05592/) (2022) - [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022) - [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022) - [Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models](https://mlsanity.com/paper/1909.11299/) (2019) - [Towards Backward-Compatible Representation Learning](https://mlsanity.com/paper/2003.11942/) (2020) - [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020) - [An Empirical Analysis of Backward Compatibility in Machine Learning Systems](https://mlsanity.com/paper/2008.04572/) (2020) - [Neural Domain Adaptation for Biomedical Question Answering](https://mlsanity.com/paper/1706.03610/) (2017) - [Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates](https://mlsanity.com/paper/2105.03048/) (2021) - [An Introduction to Lifelong Supervised Learning](https://mlsanity.com/paper/2207.04354/) (2022) ## Built on by (most cited first, 7 of 7) - [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024) - [Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion](https://mlsanity.com/paper/2302.02080/) (2023) - [DEM: Distribution Edited Model for Training with Mixed Data Distributions](https://mlsanity.com/paper/2406.15570/) (2024) - [System 1&2 Synergy via Dynamic Model Interpolation](https://mlsanity.com/paper/2601.21414/) (2026) - [Regression-aware Continual Learning for Android Malware Detection](https://mlsanity.com/paper/2507.18313/) (2025) - [Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA](https://mlsanity.com/paper/2608.22856/) (2026) - [Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlases](https://mlsanity.com/paper/2609.34478/) (2026) ## Abstract (arXiv) Backward compatibility of model predictions is a desired property when updating a machine learning driven application. It allows to seamlessly improve the underlying model without introducing regression bugs. In classification tasks these bugs occur in the form of negative flips. This means an instance that was correctly classified by the old model is now classified incorrectly by the updated model. This has direct negative impact on the user experience of such systems e.g. a frequently used voice assistant query is suddenly misclassified. A common reason to update the model is when new training data becomes available and needs to be incorporated. Simply retraining the model with the updated data introduces the unwanted negative flips. We study the problem of regression during data updates and propose Backward Compatible Weight Interpolation (BCWI). This method interpolates between the weights of the old and new model and we show in extensive experiments that it reduces negative flips without sacrificing the improved accuracy of the new model. BCWI is straight forward to implement and does not increase inference cost. We also explore the use of importance weighting during interpolation and averaging the weights of multiple new models in order to further reduce negative flips.