# Model Stability with Continuous Data Updates

arXiv: https://arxiv.org/abs/2201.05692 · HTML: https://mlsanity.com/paper/2201.05692/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 14 Jan 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals](https://mlsanity.com/paper/1809.04198/) (2018)
- [Satisfying Real-world Goals with Dataset Constraints](https://mlsanity.com/paper/1606.07558/) (2016)
- [Degrees of Freedom in Deep Neural Networks](https://mlsanity.com/paper/1603.09260/) (2016)
- [ALL-IN-1: Short Text Classification with One Model for All Languages](https://mlsanity.com/paper/1710.09589/) (2017)
- [A Comparison of Word-based and Context-based Representations for Classification Problems in Health Informatics](https://mlsanity.com/paper/1906.05468/) (2019)

## Built on by (most cited first, 9 of 9)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Backward Compatibility During Data Updates by Weight Interpolation](https://mlsanity.com/paper/2301.10546/) (2023)
- [Measuring the Instability of Fine-Tuning](https://mlsanity.com/paper/2302.07778/) (2023)
- [On the Prediction Instability of Graph Neural Networks](https://mlsanity.com/paper/2205.10070/) (2022)
- [An Epistemic and Aleatoric Decomposition of Arbitrariness to Constrain the Set of Good Models](https://mlsanity.com/paper/2302.04525/) (2023)
- [PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models](https://mlsanity.com/paper/2505.05577/) (2025)
- [Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks](https://mlsanity.com/paper/2310.00946/) (2023)
- [Stable Update of Regression Trees](https://mlsanity.com/paper/2402.13655/) (2024)
- [Revisiting the Relation Between Robustness and Universality](https://mlsanity.com/paper/2510.19427/) (2025)

## Abstract (arXiv)

In this paper, we study the "stability" of machine learning (ML) models within the context of larger, complex NLP systems with continuous training data updates. For this study, we propose a methodology for the assessment of model stability (which we refer to as jitter under various experimental conditions. We find that model design choices, including network architecture and input representation, have a critical impact on stability through experiments on four text classification tasks and two sequence labeling tasks. In classification tasks, non-RNN-based models are observed to be more stable than RNN-based ones, while the encoder-decoder model is less stable in sequence labeling tasks. Moreover, input representations based on pre-trained fastText embeddings contribute to more stability than other choices. We also show that two learning strategies -- ensemble models and incremental training -- have a significant influence on stability. We recommend ML model designers account for trade-offs in accuracy and jitter when making modeling choices.