mlsanityData to 14 Oct 2026

Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation

arXiv 2205.10868 ↗First posted 22 May 2022cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a standard component in deep reinforcement learning, is often used to reduce forgetting and improve sample efficiency by storing experiences in a large buffer and using them for training later. However, a large replay buffer results in a heavy memory burden, especially for onboard and edge devices with limited memory capacities. We propose memory-efficient reinforcement learning algorithms based on the deep Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network. Compared to baseline methods, our algorithms achieve comparable or better performance in both feature-based and image-based tasks while easing the burden of large experience replay buffers.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

56 of 79 references.

References · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
38 more
  1. 2016Progressive Neural Networkscited by 1,361
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2017Continual Learning with Deep Generative Replaycited by 1,158
  4. 2018Experience Replay for Continual Learningcited by 1,095
  5. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  7. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  8. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  9. 2019Three scenarios for continual learningcited by 614
  10. 2019Gradient based sample selection for online continual learningcited by 571
  11. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  12. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  13. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  14. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  15. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  16. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  17. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  18. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  19. 2018Selective Experience Replay for Lifelong Learningcited by 241
  20. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  21. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  22. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 179
  23. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  24. 2018Towards Robust Evaluations of Continual Learningcited by 148
  25. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  26. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  27. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  28. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  29. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 104
  30. 2017Deep Generative Dual Memory Network for Continual Learningcited by 59
  31. 2020SpaceNet: Make Free Space For Continual Learningcited by 53
  32. 2017Lifelong Generative Modelingcited by 51
  33. 2022Online Continual Learning for Embedded Devicescited by 49
  34. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  35. 2018Selfless Sequential Learningcited by 44
  36. 2019Policy Consolidation for Continual Reinforcement Learningcited by 39
  37. 2021Dual-Teacher Class-Incremental Learning With Data-Free Generative Replaycited by 31
  38. 2020Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgettingcited by 29

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 2; 2026: 1.

9 papers.

  1. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  2. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  3. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 18
  4. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 17
  5. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  6. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  7. 2025LiBOG: Lifelong Learning for Black-Box Optimizer Generationcited by 3
  8. 2025Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functionscited by 0
  9. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0