# Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation

arXiv: https://arxiv.org/abs/2109.00525 · HTML: https://mlsanity.com/paper/2109.00525/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 43 of its 69 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 43)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [State Entropy Maximization with Random Encoders for Efficient Exploration](https://mlsanity.com/paper/2102.09430/) (2021)
- [Continual Learning via Neural Pruning](https://mlsanity.com/paper/1903.04476/) (2019)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning](https://mlsanity.com/paper/1710.03937/) (2017)
- [Divide-and-Conquer Reinforcement Learning](https://mlsanity.com/paper/1711.09874/) (2017)

## Built on by (most cited first, 15 of 15)

- [Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization](https://mlsanity.com/paper/2209.00347/) (2022)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Replay-enhanced Continual Reinforcement Learning](https://mlsanity.com/paper/2311.11557/) (2023)
- [Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces](https://mlsanity.com/paper/2410.15698/) (2024)
- [Continual Knowledge Adaptation for Reinforcement Learning](https://mlsanity.com/paper/2510.19314/) (2025)
- [Matching DNN Compression and Cooperative Training with Resources and Data Availability](https://mlsanity.com/paper/2212.02304/) (2022)
- [A Study of Continual Learning Methods for Q-Learning](https://mlsanity.com/paper/2206.03934/) (2022)
- [Continual Visual Reinforcement Learning with A Life-Long World Model](https://mlsanity.com/paper/2303.06572/) (2023)
- [LiBOG: Lifelong Learning for Black-Box Optimizer Generation](https://mlsanity.com/paper/2505.13025/) (2025)
- [Cyclical Weight Consolidation: Towards Solving Catastrophic Forgetting in Serial Federated Learning](https://mlsanity.com/paper/2405.10647/) (2024)
- [Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal](https://mlsanity.com/paper/2409.02512/) (2024)
- [Reinforcement Learning Increases Wind Farm Power Production by Enabling Closed-Loop Collaborative Control](https://mlsanity.com/paper/2506.20554/) (2025)
- [Leveraging Knowledge Distillation for Efficient Deep Reinforcement Learning in Resource-Constrained Environments](https://mlsanity.com/paper/2310.10170/) (2023)
- [CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community](https://mlsanity.com/paper/2407.06485/) (2024)
- [Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs](https://mlsanity.com/paper/2605.24681/) (2026)

## Abstract (arXiv)

The powerful learning ability of deep neural networks enables reinforcement learning agents to learn competent control policies directly from continuous environments. In theory, to achieve stable performance, neural networks assume i.i.d. inputs, which unfortunately does no hold in the general reinforcement learning paradigm where the training data is temporally correlated and non-stationary. This issue may lead to the phenomenon of "catastrophic interference" and the collapse in performance. In this paper, we present IQ, i.e., interference-aware deep Q-learning, to mitigate catastrophic interference in single-task deep reinforcement learning. Specifically, we resort to online clustering to achieve on-the-fly context division, together with a multi-head network and a knowledge distillation regularization term for preserving the policy of learned contexts. Built upon deep Q networks, IQ consistently boosts the stability and performance when compared to existing methods, verified with extensive experiments on classic control and Atari tasks. The code is publicly available at: https://github.com/Sweety-dm/Interference-aware-Deep-Q-learning.