mlsanityData to 14 Oct 2026

Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation

arXiv 2109.00525 ↗First posted 1 Sep 2021cs.LG cs.AIcited by 15

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

The powerful learning ability of deep neural networks enables reinforcement learning agents to learn competent control policies directly from continuous environments. In theory, to achieve stable performance, neural networks assume i.i.d. inputs, which unfortunately does no hold in the general reinforcement learning paradigm where the training data is temporally correlated and non-stationary. This issue may lead to the phenomenon of "catastrophic interference" and the collapse in performance. In this paper, we present IQ, i.e., interference-aware deep Q-learning, to mitigate catastrophic interference in single-task deep reinforcement learning. Specifically, we resort to online clustering to achieve on-the-fly context division, together with a multi-head network and a knowledge distillation regularization term for preserving the policy of learned contexts. Built upon deep Q networks, IQ consistently boosts the stability and performance when compared to existing methods, verified with extensive experiments on classic control and Atari tasks. The code is publicly available at: https://github.com/Sweety-dm/Interference-aware-Deep-Q-learning.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 43 of its 69 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

43 of 69 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  9. 2016OpenAI Gymcited by 1,736
  10. 2016Progressive Neural Networkscited by 1,361
  11. 2015Prioritized Experience Replaycited by 1,099
  12. 2018Experience Replay for Continual Learningcited by 1,095
31 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  5. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  6. 2015Policy Distillationcited by 440
  7. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  8. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  9. 2018Selective Experience Replay for Lifelong Learningcited by 241
  10. 2020Revisiting Fundamentals of Experience Replaycited by 175
  11. 2019Continual Unsupervised Representation Learningcited by 147
  12. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  13. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  14. 2019Continual Learning via Neural Pruningcited by 101
  15. 2019Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challengescited by 93
  16. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  17. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 69
  18. 2017Divide-and-Conquer Reinforcement Learningcited by 48
  19. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  20. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  21. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  22. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 36
  23. 2020Deep Reinforcement Learning based Automatic Exploration for Navigation in Unknown Environmentcited by 34
  24. 2019Long-Range Indoor Navigation with PRM-RLcited by 31
  25. 2018The Utility of Sparse Representations for Control in Reinforcement Learningcited by 24
  26. 2019Continual Reinforcement Learning in 3D Non-stationary Environmentscited by 22
  27. 2020Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networkscited by 17
  28. 2019Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fictioncited by 14
  29. 2020On Catastrophic Interference in Atari 2600 Gamescited by 13
  30. 2020Towards a practical measure of interference for reinforcement learningcited by 6
  31. 2019Overcoming Catastrophic Interference in Online Reinforcement Learning with Dynamic Self-Organizing Mapscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 4; 2025: 3; 2026: 1.

15 papers.

Sort:

By area:

  1. 2022Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualizationcited by 14
  2. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  3. 2023Replay-enhanced Continual Reinforcement Learningcited by 7
  4. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  5. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 5
  6. 2022Matching DNN Compression and Cooperative Training with Resources and Data Availabilitycited by 4
  7. 2022A Study of Continual Learning Methods for Q-Learningcited by 3
  8. 2023Continual Visual Reinforcement Learning with A Life-Long World Modelcited by 3
  9. 2025LiBOG: Lifelong Learning for Black-Box Optimizer Generationcited by 3
  10. 2024Cyclical Weight Consolidation: Towards Solving Catastrophic Forgetting in Serial Federated Learningcited by 2
  11. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsalcited by 2
  12. 2025Reinforcement Learning Increases Wind Farm Power Production by Enabling Closed-Loop Collaborative Controlcited by 2
3 more
  1. 2023Leveraging Knowledge Distillation for Efficient Deep Reinforcement Learning in Resource-Constrained Environmentscited by 0
  2. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0
  3. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Deep Reinforcement Learning in Applied Control: Challenges, Analysis, and Insights
  2. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approach
  3. 2025InterQ: A DQN Framework for Optimal Intermittent Control
  4. 2023Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discovery
  5. 2025Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GA
  6. 2024Reinforcement Learning Approach for Integrating Compressed Contexts into Knowledge Graphs
  7. 2024Deep Reinforcement Learning: A Convex Optimization Approach
  8. 2024Intervention-Assisted Policy Gradient Methods for Online Stochastic Queuing Network Optimization: Technical Report