# On Continual Model Refinement in Out-of-Distribution Data Streams

arXiv: https://arxiv.org/abs/2205.02014 · HTML: https://mlsanity.com/paper/2205.02014/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Accelerating Deep Learning by Focusing on the Biggest Losers](https://mlsanity.com/paper/1910.00762/) (2019)
- [Continual Learning for Text Classification with Information Disentanglement Based Regularization](https://mlsanity.com/paper/2104.05489/) (2021)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization](https://mlsanity.com/paper/1907.04371/) (2019)
- [Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning](https://mlsanity.com/paper/2104.08808/) (2021)

## Built on by (most cited first, 22 of 22)

- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Fine-tuned Language Models are Continual Learners](https://mlsanity.com/paper/2205.12393/) (2022)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA](https://mlsanity.com/paper/2312.11795/) (2023)
- [Lifelong Language Pretraining with Distribution-Specialized Experts](https://mlsanity.com/paper/2305.12281/) (2023)
- [Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation](https://mlsanity.com/paper/2305.14327/) (2023)
- [Data-driven Discovery with Large Generative Models](https://mlsanity.com/paper/2402.13610/) (2024)
- [How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances](https://mlsanity.com/paper/2310.07343/) (2023)
- [Challenging Common Assumptions about Catastrophic Forgetting](https://mlsanity.com/paper/2207.04543/) (2022)
- [LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models](https://mlsanity.com/paper/2406.20030/) (2024)
- [ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks](https://mlsanity.com/paper/2405.14211/) (2024)
- [What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement](https://mlsanity.com/paper/2402.01865/) (2024)
- [Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study](https://mlsanity.com/paper/2302.04618/) (2023)
- [MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting](https://mlsanity.com/paper/2407.00875/) (2024)
- [Scalable Model Editing via Customized Expert Networks](https://mlsanity.com/paper/2404.02699/) (2024)
- [Gradient Localization Improves Lifelong Pretraining of Language Models](https://mlsanity.com/paper/2411.04448/) (2024)
- [Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning](https://mlsanity.com/paper/2408.17070/) (2024)
- [Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples](https://mlsanity.com/paper/2306.01206/) (2023)
- [Better Late Than Never: Formulating and Benchmarking Recommendation Editing](https://mlsanity.com/paper/2406.04553/) (2024)
- [Reversible Lifelong Model Editing via Semantic Routing-Based LoRA](https://mlsanity.com/paper/2603.11239/) (2026)
- [Continual Learning for Out-of-Distribution Pedestrian Detection](https://mlsanity.com/paper/2306.15117/) (2023)

## Abstract (arXiv)

Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams while overcoming catastrophic forgetting. However, existing continual learning (CL) problem setups cannot cover such a realistic and complex scenario. In response to this, we propose a new CL problem formulation dubbed continual model refinement (CMR). Compared to prior CL settings, CMR is more practical and introduces unique challenges (boundary-agnostic and non-stationary distribution shift, diverse mixtures of multiple OOD data clusters, error-centric streams, etc.). We extend several existing CL approaches to the CMR setting and evaluate them extensively. For benchmarking and analysis, we propose a general sampling algorithm to obtain dynamic OOD data streams with controllable non-stationarity, as well as a suite of metrics measuring various aspects of online performance. Our experiments and detailed analysis reveal the promise and challenges of the CMR problem, supporting that studying CMR in dynamic OOD streams can benefit the longevity of deployed NLP models in production.