# Online Continual Learning for Time Series: a Natural Score-driven Approach

arXiv: https://arxiv.org/abs/2601.12931 · HTML: https://mlsanity.com/paper/2601.12931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Builds on (most cited first, 30 of 31)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling](https://mlsanity.com/paper/2309.12659/) (2023)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [Learning Fast and Slow for Online Time Series Forecasting](https://mlsanity.com/paper/2202.11672/) (2022)
- [Online Learning for Time Series Prediction](https://mlsanity.com/paper/1302.6927/) (2013)
- [Online Natural Gradient as a Kalman Filter](https://mlsanity.com/paper/1703.00209/) (2017)
- [On the Influence of Momentum Acceleration on Online Learning](https://mlsanity.com/paper/1603.04136/) (2016)
- [A Comprehensive Empirical Evaluation on Online Continual Learning](https://mlsanity.com/paper/2308.10328/) (2023)
- [Proactive Model Adaptation Against Concept Drift for Online Time Series Forecasting](https://mlsanity.com/paper/2412.08435/) (2024)
- [Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How](https://mlsanity.com/paper/2311.04898/) (2023)
- [Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning](https://mlsanity.com/paper/2402.09542/) (2024)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Online Curvature-Aware Replay: Leveraging mathbf2nd Order Information for Online Continual Learning](https://mlsanity.com/paper/2502.01866/) (2025)

## Abstract (arXiv)

Online continual learning (OCL) methods adapt to changing environments without forgetting past knowledge. Similarly, online time series forecasting (OTSF) is a real-world problem where data evolve in time and success depends on both rapid adaptation and long-term memory. Indeed, time-varying and regime-switching forecasting models have been extensively studied, offering a strong justification for the use of OCL in these settings. Building on recent work that applies OCL to OTSF, this paper aims to strengthen the theoretical and practical connections between time series methods and OCL. First, we reframe neural network optimization as a parameter filtering problem, showing that natural gradient descent is a score-driven method and proving its information-theoretic optimality. Then, we show that using a Student's t likelihood in addition to natural gradient induces a bounded update, which improves robustness to outliers. Finally, we introduce Natural Score-driven Replay (NatSR), which combines our robust optimizer with a replay buffer and a dynamic scale heuristic that improves fast adaptation at regime drifts. Empirical results demonstrate that NatSR achieves stronger forecasting performance than more complex state-of-the-art methods.