# Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.07854 · HTML: https://mlsanity.com/paper/2106.07854/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 14 of 14)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks](https://mlsanity.com/paper/2008.03658/) (2020)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation](https://mlsanity.com/paper/1907.13223/) (2019)
- [Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware](https://mlsanity.com/paper/2003.01157/) (2020)
- [Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2009.14456/) (2020)
- [Finite Meta-Dynamic Neurons in Spiking Neural Networks for Spatio-temporal Learning](https://mlsanity.com/paper/2010.03140/) (2020)

## Built on by (most cited first, 1 of 1)

- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)

## Abstract (arXiv)

With the Deep Neural Networks (DNNs) as a powerful function approximator, Deep Reinforcement Learning (DRL) has been excellently demonstrated on robotic control tasks. Compared to DNNs with vanilla artificial neurons, the biologically plausible Spiking Neural Network (SNN) contains a diverse population of spiking neurons, making it naturally powerful on state representation with spatial and temporal information. Based on a hybrid learning framework, where a spike actor-network infers actions from states and a deep critic network evaluates the actor, we propose a Population-coding and Dynamic-neurons improved Spiking Actor Network (PDSAN) for efficient state representation from two different scales: input coding and neuronal coding. For input coding, we apply population coding with dynamically receptive fields to directly encode each input state component. For neuronal coding, we propose different types of dynamic-neurons (containing 1st-order and 2nd-order neuronal dynamics) to describe much more complex neuronal dynamics. Finally, the PDSAN is trained in conjunction with deep critic networks using the Twin Delayed Deep Deterministic policy gradient algorithm (TD3-PDSAN). Extensive experimental results show that our TD3-PDSAN model achieves better performance than state-of-the-art models on four OpenAI gym benchmark tasks. It is an important attempt to improve RL with SNN towards the effective computation satisfying biological plausibility.