# A Complementary Learning Systems Approach to Temporal Difference Learning

arXiv: https://arxiv.org/abs/1905.02636 · HTML: https://mlsanity.com/paper/1905.02636/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2019.

## Built on by (most cited first, 11 of 11)

- [Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning](https://mlsanity.com/paper/2405.02766/) (2024)
- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [Exploit CAM by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2303.02449/) (2023)
- [A Dual-Memory Architecture for Reinforcement Learning on Neuromorphic Platforms](https://mlsanity.com/paper/2103.04780/) (2021)
- [Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation](https://mlsanity.com/paper/2112.15156/) (2021)
- [AKF-SR: Adaptive Kalman Filtering-based Successor Representation](https://mlsanity.com/paper/2204.00049/) (2022)
- [Generating Explanations from Deep Reinforcement Learning Using Episodic Memory](https://mlsanity.com/paper/2205.08926/) (2022)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)
- [Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation](https://mlsanity.com/paper/2405.08969/) (2024)
- [On Neural Consolidation for Transfer in Reinforcement Learning](https://mlsanity.com/paper/2210.02240/) (2022)
- [Federated Transformed Learning for a Circular, Secure, and Tiny AI](https://mlsanity.com/paper/2311.14371/) (2023)

## Abstract (arXiv)

Complementary Learning Systems (CLS) theory suggests that the brain uses a 'neocortical' and a 'hippocampal' learning system to achieve complex behavior. These two systems are complementary in that the 'neocortical' system relies on slow learning of distributed representations while the 'hippocampal' system relies on fast learning of pattern-separated representations. Both of these systems project to the striatum, which is a key neural structure in the brain's implementation of Reinforcement Learning (RL). Current deep RL approaches share similarities with a 'neocortical' system because they slowly learn distributed representations through backpropagation in Deep Neural Networks (DNNs). An ongoing criticism of such approaches is that they are data inefficient and lack flexibility. CLS theory suggests that the addition of a 'hippocampal' system could address these criticisms. In the present study we propose a novel algorithm known as Complementary Temporal Difference Learning (CTDL), which combines a DNN with a Self-Organising Map (SOM) to obtain the benefits of both a 'neocortical' and a 'hippocampal' system. Key features of CTDL include the use of Temporal Difference (TD) error to update a SOM and the combination of a SOM and DNN to calculate action values. We evaluate CTDL on grid worlds and the Cart-Pole environment, and show several benefits over the classic Deep Q-Network (DQN) approach. These results demonstrate (1) the utility of complementary learning systems for the evaluation of actions, (2) that the TD error signal is a useful form of communication between the two systems and (3) the biological plausibility of the proposed approach.