# Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control

arXiv: https://arxiv.org/abs/2010.09635 · HTML: https://mlsanity.com/paper/2010.09635/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 30 of 49)

- [Human-Level Control through Directly-Trained Deep Spiking Q-Networks](https://mlsanity.com/paper/2201.07211/) (2021)
- [Deep Reinforcement Learning with Spiking Q-learning](https://mlsanity.com/paper/2201.09754/) (2022)
- [Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning](https://mlsanity.com/paper/2401.05444/) (2024)
- [Solving the Spike Feature Information Vanishing Problem in Spiking Deep Q Network with Potential Based Normalization](https://mlsanity.com/paper/2206.03654/) (2022)
- [Applications of Spiking Neural Networks in Visual Place Recognition](https://mlsanity.com/paper/2311.13186/) (2023)
- [Fully Spiking Neural Network for Legged Robots](https://mlsanity.com/paper/2310.05022/) (2023)
- [Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control](https://mlsanity.com/paper/2505.24161/) (2025)
- [Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network](https://mlsanity.com/paper/2301.10292/) (2022)
- [SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network](https://mlsanity.com/paper/2310.06488/) (2023)
- [Multi-compartment Neuron and Population Encoding Powered Spiking Neural Network for Deep Distributional Reinforcement Learning](https://mlsanity.com/paper/2301.07275/) (2023)
- [Noisy Spiking Actor Network for Exploration](https://mlsanity.com/paper/2403.04162/) (2024)
- [GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL](https://mlsanity.com/paper/2404.15597/) (2024)
- [Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons](https://mlsanity.com/paper/2506.03392/) (2025)
- [CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning](https://mlsanity.com/paper/2509.23791/) (2025)
- [BioGrad: Biologically Plausible Gradient-Based Learning for Spiking Neural Networks](https://mlsanity.com/paper/2110.14092/) (2021)
- [A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations](https://mlsanity.com/paper/2307.02947/) (2023)
- [Biologically-Plausible Topology Improved Spiking Actor Network for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2403.20163/) (2024)
- [Direct Training Needs Regularisation: Anytime Optimal Inference Spiking Neural Network](https://mlsanity.com/paper/2405.00699/) (2024)
- [New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles](https://mlsanity.com/paper/2512.01882/) (2025)
- [Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning](https://mlsanity.com/paper/2106.07854/) (2021)
- [Neuromorphic dreaming: A pathway to efficient learning in artificial agents](https://mlsanity.com/paper/2405.15616/) (2024)
- [ES-Parkour: Advanced Robot Parkour with Bio-inspired Event Camera and Spiking Neural Network](https://mlsanity.com/paper/2503.09985/) (2025)
- [Population-Coded Spiking Neural Networks for High-Dimensional Robotic Control](https://mlsanity.com/paper/2510.10516/) (2025)
- [Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks](https://mlsanity.com/paper/2510.24461/) (2025)
- [Dynamic Weight Adaptation in Spiking Neural Networks Inspired by Biological Homeostasis](https://mlsanity.com/paper/2511.17563/) (2025)
- [Training slow silicon neurons to control extremely fast robots with spiking reinforcement learning](https://mlsanity.com/paper/2601.21548/) (2026)
- [Error Amplification Limits ANN-to-SNN Conversion in Continuous Control](https://mlsanity.com/paper/2601.21778/) (2026)
- [SpikeVLA: Vision-Language-Action Models with Spiking Neural Networks](https://mlsanity.com/paper/2606.27807/) (2026)
- [A Novel Neuromorphic Processors Realization of Spiking Deep Reinforcement Learning for Portfolio Management](https://mlsanity.com/paper/2203.14159/) (2022)
- [EEGSN: Towards Efficient Low-latency Decoding of EEG with Graph Spiking Neural Networks](https://mlsanity.com/paper/2304.07655/) (2023)

## Abstract (arXiv)

The energy-efficient control of mobile robots is crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces, which cannot be offset by limited on-board resources. An emerging non-Von Neumann model of intelligence, where spiking neural networks (SNNs) are run on neuromorphic processors, is regarded as an energy-efficient and robust alternative to the state-of-the-art real-time robotic controllers for low dimensional control tasks. The challenge now for this new computing paradigm is to scale so that it can keep up with real-world tasks. To do so, SNNs need to overcome the inherent limitations of their training, namely the limited ability of their spiking neurons to represent information and the lack of effective learning algorithms. Here, we propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL). The population coding scheme dramatically increased the representation capacity of the network and the hybrid learning combined the training advantages of deep networks with the energy-efficient inference of spiking networks. To show the general applicability of our approach, we integrated it with a spectrum of both on-policy and off-policy DRL algorithms. We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control. To allow for a fair comparison among all methods, we validated them on OpenAI gym tasks. Our Loihi-run PopSAN consumed 140 times less energy per inference when compared against the deep actor network on Jetson TX2, and had the same level of performance. Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.