# Staying Alive: Online Neural Network Maintenance and Systemic Drift

arXiv: https://arxiv.org/abs/2503.17681 · HTML: https://mlsanity.com/paper/2503.17681/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 72 references that are in the map.
- First posted on arXiv 22 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [Efficient Continual Learning with Modular Networks and Task-Driven Priors](https://mlsanity.com/paper/2012.12631/) (2020)
- [From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors](https://mlsanity.com/paper/2203.11070/) (2022)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [A study on the plasticity of neural networks](https://mlsanity.com/paper/2106.00042/) (2021)
- [SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient](https://mlsanity.com/paper/1811.04504/) (2018)
- [Physics-constrained Deep Learning of Multi-zone Building Thermal Dynamics](https://mlsanity.com/paper/2011.05987/) (2020)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Rapid online learning and robust recall in a neuromorphic olfactory circuit](https://mlsanity.com/paper/1906.07067/) (2019)
- [DySMHO: Data-Driven Discovery of Governing Equations for Dynamical Systems via Moving Horizon Optimization](https://mlsanity.com/paper/2108.00069/) (2021)
- [Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset](https://mlsanity.com/paper/2411.04034/) (2024)
- [Automatically detecting data drift in machine learning classifiers](https://mlsanity.com/paper/2111.05672/) (2021)
- [ORFit: One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares](https://mlsanity.com/paper/2207.13853/) (2022)
- [Kalman Bayesian Neural Networks for Closed-form Online Learning](https://mlsanity.com/paper/2110.00944/) (2021)
- [Online Learning for Recommendations at Grubhub](https://mlsanity.com/paper/2107.07106/) (2021)
- [Neural Network-based Flight Control Systems: Present and Future](https://mlsanity.com/paper/2206.05596/) (2022)
- [Efficient Full-Matrix Adaptive Regularization](https://mlsanity.com/paper/1806.02958/) (2018)

## Abstract (arXiv)

We present the Subset Extended Kalman Filter (SEKF) as a method to update previously trained model weights online rather than retraining or finetuning them when the system a model represents drifts away from the conditions under which it was trained. We identify the parameters to be updated using the gradient of the loss function and use the SEKF to update only these parameters. We compare finetuning and SEKF for online model maintenance in the presence of systemic drift through four dynamic regression case studies and find that the SEKF is able to maintain model accuracy as-well if not better than finetuning while requiring significantly less time per iteration, and less hyperparameter tuning.