# Learning fast changing slow in spiking neural networks

arXiv: https://arxiv.org/abs/2402.10069 · HTML: https://mlsanity.com/paper/2402.10069/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [full-FORCE: A Target-Based Method for Training Recurrent Networks](https://mlsanity.com/paper/1710.03070/) (2017)
- [Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network](https://mlsanity.com/paper/1702.06463/) (2017)
- [Evolving Connectivity for Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2305.17650/) (2023)
- [Sleep-like slow oscillations improve visual classification through synaptic homeostasis and memory association in a thalamo-cortical model](https://mlsanity.com/paper/1810.10498/) (2018)
- [Training dynamically balanced excitatory-inhibitory networks](https://mlsanity.com/paper/1812.11424/) (2018)
- [Beyond spiking networks: the computational advantages of dendritic amplification and input segregation](https://mlsanity.com/paper/2211.02553/) (2022)
- [Error-based or target-based? A unifying framework for learning in recurrent spiking networks](https://mlsanity.com/paper/2109.01039/) (2021)
- [Towards biologically plausible Dreaming and Planning in recurrent spiking networks](https://mlsanity.com/paper/2205.10044/) (2022)

## Abstract (arXiv)

Reinforcement learning (RL) faces substantial challenges when applied to real-life problems, primarily stemming from the scarcity of available data due to limited interactions with the environment. This limitation is exacerbated by the fact that RL often demands a considerable volume of data for effective learning. The complexity escalates further when implementing RL in recurrent spiking networks, where inherent noise introduced by spikes adds a layer of difficulty. Life-long learning machines must inherently resolve the plasticity-stability paradox. Striking a balance between acquiring new knowledge and maintaining stability is crucial for artificial agents. To address this challenge, we draw inspiration from machine learning technology and introduce a biologically plausible implementation of proximal policy optimization, referred to as lf-cs (learning fast changing slow). Our approach results in two notable advancements: firstly, the capacity to assimilate new information into a new policy without requiring alterations to the current policy; and secondly, the capability to replay experiences without experiencing policy divergence. Furthermore, when contrasted with other experience replay (ER) techniques, our method demonstrates the added advantage of being computationally efficient in an online setting. We demonstrate that the proposed methodology enhances the efficiency of learning, showcasing its potential impact on neuromorphic and real-world applications.