# Biologically-Plausible Topology Improved Spiking Actor Network for Efficient Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.20163 · HTML: https://mlsanity.com/paper/2403.20163/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [Spiking and neuromorphic RL](https://mlsanity.com/deeprl/spiking-and-neuromorphic-rl/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem](https://mlsanity.com/paper/1706.10059/) (2017)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware](https://mlsanity.com/paper/2003.01157/) (2020)
- [Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2009.14456/) (2020)
- [ODE-based Recurrent Model-free Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2309.14078/) (2023)
- [Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network](https://mlsanity.com/paper/2301.10292/) (2022)
- [Attention-free Spikformer: Mixing Spike Sequences with Simple Linear Transforms](https://mlsanity.com/paper/2308.02557/) (2023)

## Built on by (most cited first, 2 of 2)

- [Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control](https://mlsanity.com/paper/2505.24161/) (2025)
- [CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning](https://mlsanity.com/paper/2509.23791/) (2025)

## Abstract (arXiv)

The success of Deep Reinforcement Learning (DRL) is largely attributed to utilizing Artificial Neural Networks (ANNs) as function approximators. Recent advances in neuroscience have unveiled that the human brain achieves efficient reward-based learning, at least by integrating spiking neurons with spatial-temporal dynamics and network topologies with biologically-plausible connectivity patterns. This integration process allows spiking neurons to efficiently combine information across and within layers via nonlinear dendritic trees and lateral interactions. The fusion of these two topologies enhances the network's information-processing ability, crucial for grasping intricate perceptions and guiding decision-making procedures. However, ANNs and brain networks differ significantly. ANNs lack intricate dynamical neurons and only feature inter-layer connections, typically achieved by direct linear summation, without intra-layer connections. This limitation leads to constrained network expressivity. To address this, we propose a novel alternative for function approximator, the Biologically-Plausible Topology improved Spiking Actor Network (BPT-SAN), tailored for efficient decision-making in DRL. The BPT-SAN incorporates spiking neurons with intricate spatial-temporal dynamics and introduces intra-layer connections, enhancing spatial-temporal state representation and facilitating more precise biological simulations. Diverging from the conventional direct linear weighted sum, the BPT-SAN models the local nonlinearities of dendritic trees within the inter-layer connections. For the intra-layer connections, the BPT-SAN introduces lateral interactions between adjacent neurons, integrating them into the membrane potential formula to ensure accurate spike firing.