# Statistical Context Detection for Deep Lifelong Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.19047 · HTML: https://mlsanity.com/paper/2405.19047/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 36 of its 61 references that are in the map.
- First posted on arXiv 29 May 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Generalized Sliced Wasserstein Distances](https://mlsanity.com/paper/1902.00434/) (2019)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization](https://mlsanity.com/paper/1802.01569/) (2018)
- [Sample Complexity of Multi-task Reinforcement Learning](https://mlsanity.com/paper/1309.6821/) (2013)
- [Same State, Different Task: Continual Reinforcement Learning without Interference](https://mlsanity.com/paper/2106.02940/) (2021)
- [Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning](https://mlsanity.com/paper/2303.09483/) (2023)
- [A statistical framework for efficient out of distribution detection in deep neural networks](https://mlsanity.com/paper/2102.12967/) (2021)
- [Wasserstein Task Embedding for Measuring Task Similarities](https://mlsanity.com/paper/2208.11726/) (2022)
- [Task-Free Continual Learning via Online Discrepancy Distance Learning](https://mlsanity.com/paper/2210.06579/) (2022)
- [Lifelong Reinforcement Learning with Modulating Masks](https://mlsanity.com/paper/2212.11110/) (2022)
- [Continual Learning Based on OOD Detection and Task Masking](https://mlsanity.com/paper/2203.09450/) (2022)
- [Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detection](https://mlsanity.com/paper/2105.09452/) (2021)

## Built on by (most cited first, 7 of 7)

- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedback](https://mlsanity.com/paper/2603.04029/) (2026)
- [Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems](https://mlsanity.com/paper/2506.05577/) (2025)
- [Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces](https://mlsanity.com/paper/2506.05702/) (2025)
- [Formalizing Task-Space Complexity for Zero-Shot Generalization](https://mlsanity.com/paper/2606.20967/) (2026)
- [Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.05298/) (2026)
- [How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning](https://mlsanity.com/paper/2610.03119/) (2026)

## Abstract (arXiv)

Context detection involves labeling segments of an online stream of data as belonging to different tasks. Task labels are used in lifelong learning algorithms to perform consolidation or other procedures that prevent catastrophic forgetting. Inferring task labels from online experiences remains a challenging problem. Most approaches assume finite and low-dimension observation spaces or a preliminary training phase during which task labels are learned. Moreover, changes in the transition or reward functions can be detected only in combination with a policy, and therefore are more difficult to detect than changes in the input distribution. This paper presents an approach to learning both policies and labels in an online deep reinforcement learning setting. The key idea is to use distance metrics, obtained via optimal transport methods, i.e., Wasserstein distance, on suitable latent action-reward spaces to measure distances between sets of data points from past and current streams. Such distances can then be used for statistical tests based on an adapted Kolmogorov-Smirnov calculation to assign labels to sequences of experiences. A rollback procedure is introduced to learn multiple policies by ensuring that only the appropriate data is used to train the corresponding policy. The combination of task detection and policy deployment allows for the optimization of lifelong reinforcement learning agents without an oracle that provides task labels. The approach is tested using two benchmarks and the results show promising performance when compared with related context detection algorithms. The results suggest that optimal transport statistical methods provide an explainable and justifiable procedure for online context detection and reward optimization in lifelong reinforcement learning.