mlsanityData to 13 Oct 2026

Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning

arXiv 2401.05444 ↗First posted 9 Jan 2024cs.NE cs.AI cs.LGcited by 10

On the map: Deep reinforcement learning methods → Spiking and neuromorphic RL

Abstract (arXiv)

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this paper, we focus on the task where the agent needs to learn multi-dimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multi-layer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based RL methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully-connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network without any floating-point matrix operations, we draw inspiration from the non-spiking interneurons found in insects and employ the membrane voltage of the non-spiking neurons to represent the action. Before the non-spiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intra-layer connections are used in output populations to enhance the representation capacity. Finally, we propose a fully spiking actor network with intra-layer connections (ILC-SAN).

Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 61 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

23 of 61 references.

References · 23

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  2. 2015Continuous control with deep reinforcement learningcited by 3,290
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  4. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  5. 2021Deep Residual Learning in Spiking Neural Networkscited by 329
  6. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 205
  7. 2018Long short-term memory and learning-to-learn in networks of spiking neuronscited by 180
  8. 2019The Heidelberg spiking datasets for the systematic evaluation of spiking neural networkscited by 137
  9. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  10. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 114
  11. 2020Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networkscited by 98
  12. 2018Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rulescited by 64
11 more
  1. 2015Memory and information processing in neuromorphic systemscited by 48
  2. 2020Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Controlcited by 44
  3. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  4. 2019Spike-Train Level Backpropagation for Training Deep Recurrent Spiking Neural Networkscited by 32
  5. 2020Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardwarecited by 32
  6. 2021A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardwarecited by 32
  7. 2021Human-Level Control through Directly-Trained Deep Spiking Q-Networkscited by 23
  8. 2020Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networkscited by 20
  9. 2019Event-based attention and tracking on neuromorphic hardwarecited by 14
  10. 2022Solving the Spike Feature Information Vanishing Problem in Spiking Deep Q Network with Potential Based Normalizationcited by 9
  11. 2020An Astrocyte-Modulated Neuromorphic Central Pattern Generator for Hexapod Robot Locomotion on Intel's Loihicited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 4.

10 papers.

  1. 2025Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Controlcited by 4
  2. 2024Noisy Spiking Actor Network for Explorationcited by 2
  3. 2024Brain-inspired Action Generation with Spiking Transformer Diffusion Policy Modelcited by 1
  4. 2026Error Amplification Limits ANN-to-SNN Conversion in Continuous Controlcited by 1
  5. 2025An Interpretable Neural Control Network with Adaptable Online Learning for Sample Efficient Robot Locomotion Learningcited by 0
  6. 2025SpikingSoft: A Spiking Neuron Controller for Bio-inspired Locomotion with Soft Snake Robotscited by 0
  7. 2025AOLO: Analysis and Optimization For Low-Carbon Oriented Wireless Large Language Model Servicescited by 0
  8. 2026Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networkscited by 0
  9. 2026NeuroTrain: Surveying Local Learning Rules for Spiking Neural Networks with an Open Benchmarking Frameworkcited by 0
  10. 2026Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computingcited by 0