# Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

arXiv: https://arxiv.org/abs/2103.12593 · HTML: https://mlsanity.com/paper/2103.12593/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 12 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks](https://mlsanity.com/paper/2009.08378/) (2020)
- [A neural attention model for speech command recognition](https://mlsanity.com/paper/1808.08929/) (2018)
- [Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks](https://mlsanity.com/paper/2005.11633/) (2020)
- [Brain-Inspired Learning on Neuromorphic Substrates](https://mlsanity.com/paper/2010.11931/) (2020)
- [STDP-based spiking deep convolutional neural networks for object recognition](https://mlsanity.com/paper/1611.01421/) (2016)
- [Low-activity supervised convolutional spiking neural networks applied to speech commands recognition](https://mlsanity.com/paper/2011.06846/) (2020)
- [TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices](https://mlsanity.com/paper/2008.04245/) (2020)
- [Multi-layered Spiking Neural Network with Target Timestamp Threshold Adaptation and STDP](https://mlsanity.com/paper/1904.01908/) (2019)

## Built on by (most cited first, 30 of 115)

- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Attention Spiking Neural Networks](https://mlsanity.com/paper/2209.13929/) (2022)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline](https://mlsanity.com/paper/2304.06793/) (2023)
- [Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings](https://mlsanity.com/paper/2306.17670/) (2023)
- [Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time](https://mlsanity.com/paper/2112.11231/) (2021)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://mlsanity.com/paper/2406.03287/) (2024)
- [SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking](https://mlsanity.com/paper/2407.04752/) (2024)
- [STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks](https://mlsanity.com/paper/2210.05241/) (2022)
- [Learning Long Sequences in Spiking Neural Networks](https://mlsanity.com/paper/2401.00955/) (2023)
- [SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models](https://mlsanity.com/paper/2408.14909/) (2024)
- [Spiker+: a framework for the generation of efficient Spiking Neural Networks FPGA accelerators for inference at the edge](https://mlsanity.com/paper/2401.01141/) (2024)
- [Loss shaping enhances exact gradient learning with Eventprop in spiking neural networks](https://mlsanity.com/paper/2212.01232/) (2022)
- [Inherent Redundancy in Spiking Neural Networks](https://mlsanity.com/paper/2308.08227/) (2023)
- [Co-learning synaptic delays, weights and adaptation in spiking neural networks](https://mlsanity.com/paper/2311.16112/) (2023)
- [Balanced Resonate-and-Fire Neurons](https://mlsanity.com/paper/2402.14603/) (2024)
- [Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning](https://mlsanity.com/paper/2401.05444/) (2024)
- [Temporal Knowledge Sharing enable Spiking Neural Network Learning from Past and Future](https://mlsanity.com/paper/2304.06540/) (2023)
- [FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks](https://mlsanity.com/paper/2501.14744/) (2024)
- [DelGrad: Exact event-based gradients for training delays and weights on spiking neuromorphic hardware](https://mlsanity.com/paper/2404.19165/) (2024)
- [SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Network](https://mlsanity.com/paper/2405.14398/) (2024)
- [An Energy-Efficient Spiking Neural Network for Finger Velocity Decoding for Implantable Brain-Machine Interface](https://mlsanity.com/paper/2210.06287/) (2022)
- [Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks](https://mlsanity.com/paper/2402.11984/) (2024)
- [Autaptic Synaptic Circuit Enhances Spatio-temporal Predictive Learning of Spiking Neural Networks](https://mlsanity.com/paper/2406.00405/) (2024)
- [Ternary Spike-based Neuromorphic Signal Processing System](https://mlsanity.com/paper/2407.05310/) (2024)

## Abstract (arXiv)

Inspired by more detailed modeling of biological neurons, Spiking neural networks (SNNs) have been investigated both as more biologically plausible and potentially more powerful models of neural computation, and also with the aim of extracting biological neurons' energy efficiency; the performance of such networks however has remained lacking compared to classical artificial neural networks (ANNs). Here, we demonstrate how a novel surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields state-of-the-art for SNNs on challenging benchmarks in the time-domain, like speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks (RNNs) and approaches that of the best modern ANNs. As these SNNs exhibit sparse spiking, we show that they theoretically are one to three orders of magnitude more computationally efficient compared to RNNs with comparable performance. Together, this positions SNNs as an attractive solution for AI hardware implementations.