# Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset

arXiv: https://arxiv.org/abs/2411.04034 · HTML: https://mlsanity.com/paper/2411.04034/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2010.14498/) (2020)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [Continual Backprop: Stochastic Gradient Descent with Persistent Randomness](https://mlsanity.com/paper/2108.06325/) (2021)
- [Maintaining Plasticity in Continual Learning via Regenerative Regularization](https://mlsanity.com/paper/2308.11958/) (2023)
- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Streaming Variational Bayes](https://mlsanity.com/paper/1307.6769/) (2013)
- [A study on the plasticity of neural networks](https://mlsanity.com/paper/2106.00042/) (2021)
- [Dynamical Models and Tracking Regret in Online Convex Programming](https://mlsanity.com/paper/1301.1254/) (2013)
- [MetaGrad: Adaptation using Multiple Learning Rates in Online Learning](https://mlsanity.com/paper/2102.06622/) (2021)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [Kalman Filter for Online Classification of Non-Stationary Data](https://mlsanity.com/paper/2306.08448/) (2023)
- [Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes](https://mlsanity.com/paper/2111.04558/) (2021)

## Built on by (most cited first, 17 of 17)

- [Plasticity Loss in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2411.04832/) (2024)
- [Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents](https://mlsanity.com/paper/2510.13704/) (2025)
- [Forget Forgetting: Continual Learning in a World of Abundant Memory](https://mlsanity.com/paper/2502.07274/) (2025)
- [A unifying framework for generalised Bayesian online learning in non-stationary environments](https://mlsanity.com/paper/2411.10153/) (2024)
- [Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streaming](https://mlsanity.com/paper/2504.09906/) (2025)
- [Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning](https://mlsanity.com/paper/2504.17490/) (2025)
- [Preserving Plasticity in Continual Learning with Adaptive Linearity Injection](https://mlsanity.com/paper/2505.09486/) (2025)
- [Is Exploration or Optimization the Problem for Deep Reinforcement Learning?](https://mlsanity.com/paper/2508.01329/) (2025)
- [Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networks](https://mlsanity.com/paper/2604.09028/) (2026)
- [Staying Alive: Online Neural Network Maintenance and Systemic Drift](https://mlsanity.com/paper/2503.17681/) (2025)
- [Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction](https://mlsanity.com/paper/2510.15422/) (2025)
- [Emotion-Inspired Learning Signals (EILS): A Homeostatic Framework for Adaptive Autonomous Agents](https://mlsanity.com/paper/2512.22200/) (2025)
- [Rethinking Plasticity in Deep Reinforcement Learning](https://mlsanity.com/paper/2603.21173/) (2026)
- [Forager: a lightweight testbed for continual learning with partial observability in RL](https://mlsanity.com/paper/2605.01131/) (2026)
- [Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization](https://mlsanity.com/paper/2605.30059/) (2026)
- [PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networks](https://mlsanity.com/paper/2607.17922/) (2026)
- [Online Learning via Learned Latent Bayesian Tracking](https://mlsanity.com/paper/2609.31559/) (2026)

## Abstract (arXiv)

Neural networks are traditionally trained under the assumption that data come from a stationary distribution. However, settings which violate this assumption are becoming more popular; examples include supervised learning under distributional shifts, reinforcement learning, continual learning and non-stationary contextual bandits. In this work we introduce a novel learning approach that automatically models and adapts to non-stationarity, via an Ornstein-Uhlenbeck process with an adaptive drift parameter. The adaptive drift tends to draw the parameters towards the initialisation distribution, so the approach can be understood as a form of soft parameter reset. We show empirically that our approach performs well in non-stationary supervised and off-policy reinforcement learning settings.