# Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning

arXiv: https://arxiv.org/abs/2109.01905 · HTML: https://mlsanity.com/paper/2109.01905/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 14 of 14)

- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://mlsanity.com/paper/1602.02830/) (2016)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences](https://mlsanity.com/paper/2005.02183/) (2020)
- [BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations](https://mlsanity.com/paper/2002.06517/) (2020)
- [Recurrent Dropout without Memory Loss](https://mlsanity.com/paper/1603.05118/) (2016)
- [Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks](https://mlsanity.com/paper/1607.02241/) (2016)

## Built on by (most cited first, 27 of 27)

- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies](https://mlsanity.com/paper/2304.12760/) (2023)
- [Spiking PointNet: Spiking Neural Networks for Point Clouds](https://mlsanity.com/paper/2310.06232/) (2023)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities](https://mlsanity.com/paper/2211.10754/) (2022)
- [Shrinking Your TimeStep: Towards Low-Latency Neuromorphic Object Recognition with Spiking Neural Network](https://mlsanity.com/paper/2401.01912/) (2024)
- [A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception](https://mlsanity.com/paper/2303.14176/) (2023)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks](https://mlsanity.com/paper/2406.07862/) (2024)
- [S-TLLR: STDP-inspired Temporal Local Learning Rule for Spiking Neural Networks](https://mlsanity.com/paper/2306.15220/) (2023)
- [Fully Spiking Neural Networks for Unified Frame-Event Object Tracking](https://mlsanity.com/paper/2505.20834/) (2025)
- [Intelligent Sensing-to-Action for Robust Autonomy at the Edge: Opportunities and Challenges](https://mlsanity.com/paper/2502.02692/) (2025)
- [Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition](https://mlsanity.com/paper/2212.01187/) (2022)
- [Multi-compartment Neuron and Population Encoding Powered Spiking Neural Network for Deep Distributional Reinforcement Learning](https://mlsanity.com/paper/2301.07275/) (2023)
- [Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology](https://mlsanity.com/paper/2312.01213/) (2023)
- [GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL](https://mlsanity.com/paper/2404.15597/) (2024)
- [Benchmarking Spiking Neural Network Learning Methods with Varying Locality](https://mlsanity.com/paper/2402.01782/) (2024)
- [Temporal Reversal Regularization for Spiking Neural Networks: Hybrid Spatio-Temporal Invariance for Generalization](https://mlsanity.com/paper/2408.09108/) (2024)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers](https://mlsanity.com/paper/2510.07924/) (2025)
- [Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model](https://mlsanity.com/paper/2305.16340/) (2023)
- [Understanding the Functional Roles of Modelling Components in Spiking Neural Networks](https://mlsanity.com/paper/2403.16674/) (2024)
- [A 71.2-μW Speech Recognition Accelerator with Recurrent Spiking Neural Network](https://mlsanity.com/paper/2503.21337/) (2025)
- [IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing](https://mlsanity.com/paper/2507.07396/) (2025)
- [Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency](https://mlsanity.com/paper/2401.10843/) (2024)
- [Gated Parametric Neuron for Spike-based Audio Recognition](https://mlsanity.com/paper/2412.01087/) (2024)
- [Stable Spike: Dual Consistency Optimization via Bitwise AND Operations for Spiking Neural Networks](https://mlsanity.com/paper/2603.11676/) (2026)

## Abstract (arXiv)

Spiking neural networks (SNNs) with leaky integrate and fire (LIF) neurons, can be operated in an event-driven manner and have internal states to retain information over time, providing opportunities for energy-efficient neuromorphic computing, especially on edge devices. Note, however, many representative works on SNNs do not fully demonstrate the usefulness of their inherent recurrence (membrane potentials retaining information about the past) for sequential learning. Most of the works train SNNs to recognize static images by artificially expanded input representation in time through rate coding. We show that SNNs can be trained for sequential tasks and propose modifications to a network of LIF neurons that enable internal states to learn long sequences and make their inherent recurrence resilient to the vanishing gradient problem. We then develop a training scheme to train the proposed SNNs with improved inherent recurrence dynamics. Our training scheme allows spiking neurons to produce multi-bit outputs (as opposed to binary spikes) which help mitigate the mismatch between a derivative of spiking neurons' activation function and a surrogate derivative used to overcome spiking neurons' non-differentiability. Our experimental results indicate that the proposed SNN architecture on TIMIT and LibriSpeech 100h dataset yields accuracy comparable to that of LSTMs (within 1.10% and 0.36%, respectively), but with 2x fewer parameters than LSTMs. The sparse SNN outputs also lead to 10.13x and 11.14x savings in multiplication operations compared to GRUs, which is generally con-sidered as a lightweight alternative to LSTMs, on TIMIT and LibriSpeech 100h datasets, respectively.