mlsanityData to 7 Oct 2026

Prediction and Control in Continual Reinforcement Learning

arXiv 2312.11669 ↗First posted 18 Dec 2023cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Temporal difference (TD) learning is often used to update the estimate of the value function which is used by RL agents to extract useful policies. In this paper, we focus on value function estimation in continual reinforcement learning. We propose to decompose the value function into two components which update at different timescales: a permanent value function, which holds general knowledge that persists over time, and a transient value function, which allows quick adaptation to new situations. We establish theoretical results showing that our approach is well suited for continual learning and draw connections to the complementary learning systems (CLS) theory from neuroscience. Empirically, this approach improves performance significantly on both prediction and control problems.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 25 of its 52 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

25 of 52 references.

References · 25

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2018Experience Replay for Continual Learningcited by 1,074
  4. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  5. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  6. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  8. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  9. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  10. 2019Meta-Learning Representations for Continual Learningcited by 189
  11. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  12. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
13 more
  1. 2023A Definition of Continual Reinforcement Learningcited by 98
  2. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  3. 2021Bootstrapped Meta-Learningcited by 45
  4. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  5. 2018Continual Reinforcement Learning with Complex Synapsescited by 30
  6. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 20
  7. 2020Jelly Bean World: A Testbed for Never-Ending Learningcited by 19
  8. 2019Meta-descent for Online, Continual Predictioncited by 10
  9. 2020Continual Reinforcement Learning with Multi-Timescale Replaycited by 10
  10. 2019Adaptive Online Planning for Continual Lifelong Learningcited by 7
  11. 2022Self-Activating Neural Ensembles for Continual Reinforcement Learningcited by 5
  12. 2023Utility-based Perturbed Gradient Descent: An Optimizer for Continual Learningcited by 5
  13. 2022Neural Distillation as a State Representation Bottleneck in Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 3.

19 papers.

Sort:

By area:

  1. 2025A Survey of Continual Reinforcement Learningcited by 16
  2. 2024Three Dogmas of Reinforcement Learningcited by 14
  3. 2024Learning Successor Features the Simple Waycited by 10
  4. 2024Position: Lifetime tuning is incompatible with continual reinforcement learningcited by 9
  5. 2024Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theorycited by 7
  6. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  7. 2025Knowledge Retention for Continual Model-Based Reinforcement Learningcited by 6
  8. 2024Investigating the Interplay of Prioritized Replay and Generalizationcited by 5
  9. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 3
  10. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsalcited by 2
  11. 2025The Cell Must Go On: Agar.io for Continual Reinforcement Learningcited by 2
  12. 2025Memory Allocation in Resource-Constrained Reinforcement Learningcited by 2
7 more
  1. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  2. 2026Position: Deployed Reinforcement Learning should be Continualcited by 1
  3. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learningcited by 0
  4. 2025Requirements for Recognition and Rapid Response to Unfamiliar Events Outside of Agent Design Scopecited by 0
  5. 2025Towards better dense rewards in Reinforcement Learning Applicationscited by 0
  6. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  7. 2026Forager: a lightweight testbed for continual learning with partial observability in RLcited by 0