# Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks

arXiv: https://arxiv.org/abs/2601.21572 · HTML: https://mlsanity.com/paper/2601.21572/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [Spiking and neuromorphic RL](https://mlsanity.com/deeprl/spiking-and-neuromorphic-rl/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [An elementary introduction to information geometry](https://mlsanity.com/paper/1808.08271/) (2018)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time](https://mlsanity.com/paper/2112.11231/) (2021)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients](https://mlsanity.com/paper/1712.06563/) (2017)
- [Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning](https://mlsanity.com/paper/2401.05444/) (2024)
- [Evolving Connectivity for Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2305.17650/) (2023)
- [Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control](https://mlsanity.com/paper/2505.24161/) (2025)
- [Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks](https://mlsanity.com/paper/2308.12063/) (2023)

## Built on by (most cited first, 1 of 1)

- [Training Non-Differentiable Networks via Optimal Transport](https://mlsanity.com/paper/2605.01928/) (2026)

## Abstract (arXiv)

Recurrent spiking neural networks (RSNNs) are a promising substrate for energy-efficient control policies, but training them for high-dimensional, long-horizon reinforcement learning remains challenging. Population-based, gradient-free optimization circumvents backpropagation through non-differentiable spike dynamics by estimating gradients. However, with finite populations, high variance of these estimates can induce harmful and overly aggressive update steps. Inspired by trust-region methods in reinforcement learning that constrain policy updates in distribution space, we propose Signal-Adaptive Trust Regions (SATR), a distributional update rule that constrains relative change by bounding KL divergence normalized by an estimated signal energy. SATR automatically expands the trust region under strong signals and contracts it when updates are noise-dominated. We instantiate SATR for Bernoulli connectivity distributions, which have shown strong empirical performance for RSNN optimization. Across a suite of high-dimensional continuous-control benchmarks, SATR improves stability under limited populations and reaches competitive returns against strong baselines including PPO-LSTM. In addition, to make SATR practical at scale, we introduce a bitset implementation for binary spiking and binary weights, substantially reducing wall-clock training time and enabling fast RSNN policy search.