# Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks

arXiv: https://arxiv.org/abs/2510.24461 · HTML: https://mlsanity.com/paper/2510.24461/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [Spiking and neuromorphic RL](https://mlsanity.com/deeprl/spiking-and-neuromorphic-rl/)

## Builds on (most cited first, 18 of 18)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Unsupervised Predictive Memory in a Goal-Directed Agent](https://mlsanity.com/paper/1803.10760/) (2018)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware](https://mlsanity.com/paper/2003.01157/) (2020)
- [Learning to Fly in Seconds](https://mlsanity.com/paper/2311.13081/) (2023)
- [Fully neuromorphic vision and control for autonomous drone flight](https://mlsanity.com/paper/2303.08778/) (2023)
- [Human-Level Control through Directly-Trained Deep Spiking Q-Networks](https://mlsanity.com/paper/2201.07211/) (2021)
- [Deep Reinforcement Learning with Spiking Q-learning](https://mlsanity.com/paper/2201.09754/) (2022)
- [Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks](https://mlsanity.com/paper/2404.14964/) (2024)
- [Evolving Connectivity for Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2305.17650/) (2023)
- [Neuromorphic Attitude Estimation and Control](https://mlsanity.com/paper/2411.13945/) (2024)

## Built on by (most cited first, 1 of 1)

- [SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers](https://mlsanity.com/paper/2608.13702/) (2026)

## Abstract (arXiv)

Neuromorphic computing systems are set to revolutionize energy-constrained robotics by achieving orders-of-magnitude efficiency gains, while enabling native temporal processing. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application to complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the stateful dynamics of SNNs require training on sequences, which in reinforcement learning (RL) is hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period. We address these challenges by systematically analyzing surrogate gradient slope settings, showing that shallower slopes increase gradient magnitude in deeper layers but reduce alignment with true gradients. In supervised learning, we find no clear preference for fixed or scheduled slopes. The effect is much more pronounced in RL settings, where shallower slopes or scheduled slopes lead to a 2.1x improvement in both training and final deployed performance. Next, we propose a novel training approach that leverages a privileged guiding policy to bootstrap the learning process, while still exploiting online environment interactions with the spiking policy. Combining our method with an adaptive slope schedule for a real-world drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including Behavioral Cloning and TD3BC, which achieve at most --200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradient learning in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.