# Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN

arXiv: https://arxiv.org/abs/2403.03409 · HTML: https://mlsanity.com/paper/2403.03409/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 73 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks](https://mlsanity.com/paper/2002.10085/) (2020)
- [Methods for Pruning Deep Neural Networks](https://mlsanity.com/paper/2011.00241/) (2020)
- [STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition](https://mlsanity.com/paper/1710.04734/) (2017)
- [Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization](https://mlsanity.com/paper/1911.00822/) (2019)
- [Pruning of Deep Spiking Neural Networks through Gradient Rewiring](https://mlsanity.com/paper/2105.04916/) (2021)
- [Lyapunov spectra of chaotic recurrent neural networks](https://mlsanity.com/paper/2006.02427/) (2020)
- [On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools](https://mlsanity.com/paper/2006.14123/) (2020)
- [A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/2104.10719/) (2021)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks](https://mlsanity.com/paper/2306.03693/) (2023)
- [Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networks](https://mlsanity.com/paper/2104.12528/) (2021)
- [Workload-Balanced Pruning for Sparse Spiking Neural Networks](https://mlsanity.com/paper/2302.06746/) (2023)
- [SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning](https://mlsanity.com/paper/2207.03677/) (2022)
- [Heterogeneous Recurrent Spiking Neural Network for Spatio-Temporal Classification](https://mlsanity.com/paper/2211.04297/) (2022)
- [Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training](https://mlsanity.com/paper/2204.11218/) (2022)
- [Characterization of Generalizability of Spike Timing Dependent Plasticity trained Spiking Neural Networks](https://mlsanity.com/paper/2105.14677/) (2021)
- [Heterogeneous Neuronal and Synaptic Dynamics for Spike-Efficient Unsupervised Learning: Theory and Design Principles](https://mlsanity.com/paper/2302.11618/) (2023)
- [GRASS: Graph Spectral Sparsification Leveraging Scalable Spectral Perturbation Analysis](https://mlsanity.com/paper/1911.04382/) (2019)
- [Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks](https://mlsanity.com/paper/2211.12714/) (2022)
- [Using noise to probe recurrent neural network structure and prune synapses](https://mlsanity.com/paper/2011.07334/) (2020)
- [Brain-Inspired Spiking Neural Network for Online Unsupervised Time Series Prediction](https://mlsanity.com/paper/2304.04697/) (2023)
- [Spectral Measurement Sparsification for Pose-Graph SLAM](https://mlsanity.com/paper/2203.13897/) (2022)

## Built on by (most cited first, 10 of 10)

- [Rethinking Spiking Neural Networks from an Ensemble Learning Perspective](https://mlsanity.com/paper/2502.14218/) (2025)
- [Topological Representations of Heterogeneous Learning Dynamics of Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2403.12462/) (2024)
- [Exploiting Heterogeneity in Timescales for Sparse Recurrent Spiking Neural Networks for Energy-Efficient Edge Computing](https://mlsanity.com/paper/2407.06452/) (2024)
- [Temporal Reversal Regularization for Spiking Neural Networks: Hybrid Spatio-Temporal Invariance for Generalization](https://mlsanity.com/paper/2408.09108/) (2024)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models](https://mlsanity.com/paper/2405.00433/) (2024)
- [A Dynamical Systems-Inspired Pruning Strategy for Addressing Oversmoothing in Graph Neural Networks](https://mlsanity.com/paper/2412.07243/) (2024)
- [FLAMES: A Hybrid Spiking-State Space Model for Adaptive Memory Retention in Event-Based Learning](https://mlsanity.com/paper/2504.01257/) (2025)
- [A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanism](https://mlsanity.com/paper/2409.12691/) (2024)
- [SpikeSMOKE: Spiking Neural Networks for Monocular 3D Object Detection with Cross-Scale Gated Coding](https://mlsanity.com/paper/2506.07737/) (2025)

## Abstract (arXiv)

Recurrent Spiking Neural Networks (RSNNs) have emerged as a computationally efficient and brain-inspired learning model. The design of sparse RSNNs with fewer neurons and synapses helps reduce the computational complexity of RSNNs. Traditionally, sparse SNNs are obtained by first training a dense and complex SNN for a target task, and, then, pruning neurons with low activity (activity-based pruning) while maintaining task performance. In contrast, this paper presents a task-agnostic methodology for designing sparse RSNNs by pruning a large randomly initialized model. We introduce a novel Lyapunov Noise Pruning (LNP) algorithm that uses graph sparsification methods and utilizes Lyapunov exponents to design a stable sparse RSNN from a randomly initialized RSNN. We show that the LNP can leverage diversity in neuronal timescales to design a sparse Heterogeneous RSNN (HRSNN). Further, we show that the same sparse HRSNN model can be trained for different tasks, such as image classification and temporal prediction. We experimentally show that, in spite of being task-agnostic, LNP increases computational efficiency (fewer neurons and synapses) and prediction performance of RSNNs compared to traditional activity-based pruning of trained dense models.