# Continual Reinforcement Learning with Complex Synapses

arXiv: https://arxiv.org/abs/1802.07239 · HTML: https://mlsanity.com/paper/1802.07239/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2018.

## Built on by (most cited first, 30 of 32)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Disentangling Transfer in Continual Reinforcement Learning](https://mlsanity.com/paper/2209.13900/) (2022)
- [Continuous Coordination As a Realistic Scenario for Lifelong Learning](https://mlsanity.com/paper/2103.03216/) (2021)
- [Block Contextual MDPs for Continual Learning](https://mlsanity.com/paper/2110.06972/) (2021)
- [Lifelong Reinforcement Learning with Modulating Masks](https://mlsanity.com/paper/2212.11110/) (2022)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning](https://mlsanity.com/paper/2205.10787/) (2022)
- [Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals](https://mlsanity.com/paper/2203.11560/) (2022)
- [Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges](https://mlsanity.com/paper/2205.14495/) (2022)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Meta-Learning Bidirectional Update Rules](https://mlsanity.com/paper/2104.04657/) (2021)
- [Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories](https://mlsanity.com/paper/2304.13424/) (2023)
- [Online Reinforcement Learning in Non-Stationary Context-Driven Environments](https://mlsanity.com/paper/2302.02182/) (2023)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Theories of synaptic memory consolidation and intelligent plasticity for continual learning](https://mlsanity.com/paper/2405.16922/) (2024)
- [The Surprising Positive Knowledge Transfer in Continual 3D Object Shape Reconstruction](https://mlsanity.com/paper/2101.07295/) (2021)
- [Current State and Future Directions for Learning in Biological Recurrent Neural Networks: A Perspective Piece](https://mlsanity.com/paper/2105.05382/) (2021)
- [Demystifying Reinforcement Learning in Time-Varying Systems](https://mlsanity.com/paper/2201.05560/) (2022)
- [Lifelong Reinforcement Learning via Neuromodulation](https://mlsanity.com/paper/2408.08446/) (2024)
- [Online Learning with Radial Basis Function Networks](https://mlsanity.com/paper/2103.08414/) (2021)
- [A Multi-Agent Reinforcement Learning Testbed for Cognitive Radio Applications](https://mlsanity.com/paper/2410.21521/) (2024)
- [Continual Knowledge Updating in LLM Systems: Learning Through Multi-Timescale Memory Dynamics](https://mlsanity.com/paper/2605.05097/) (2026)
- [Context selectivity with dynamic availability enables lifelong continual learning](https://mlsanity.com/paper/2306.01690/) (2023)
- [What Neuroscience Can Teach AI About Learning in Continuously Changing Environments](https://mlsanity.com/paper/2507.02103/) (2025)
- [Bellman Memory Units: A neuromorphic framework for synaptic reinforcement learning with an evolving network topology](https://mlsanity.com/paper/2511.16066/) (2025)
- [Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning](https://mlsanity.com/paper/2603.00903/) (2026)

## Abstract (arXiv)

Unlike humans, who are capable of continual learning over their lifetimes, artificial neural networks have long been known to suffer from a phenomenon known as catastrophic forgetting, whereby new learning can lead to abrupt erasure of previously acquired knowledge. Whereas in a neural network the parameters are typically modelled as scalar values, an individual synapse in the brain comprises a complex network of interacting biochemical components that evolve at different timescales. In this paper, we show that by equipping tabular and deep reinforcement learning agents with a synaptic model that incorporates this biological complexity (Benna & Fusi, 2016), catastrophic forgetting can be mitigated at multiple timescales. In particular, we find that as well as enabling continual learning across sequential training of two simple tasks, it can also be used to overcome within-task forgetting by reducing the need for an experience replay database.