# Pruning of Deep Spiking Neural Networks through Gradient Rewiring

arXiv: https://arxiv.org/abs/2105.04916 · HTML: https://mlsanity.com/paper/2105.04916/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 13 of 13)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Deep Rewiring: Training very sparse deep networks](https://mlsanity.com/paper/1711.05136/) (2017)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition](https://mlsanity.com/paper/1710.04734/) (2017)
- [Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization](https://mlsanity.com/paper/1911.00822/) (2019)
- [Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines](https://mlsanity.com/paper/1511.04484/) (2015)
- [Network Plasticity as Bayesian Inference](https://mlsanity.com/paper/1504.05143/) (2015)
- [Spiking neural networks trained with backpropagation for low power neuromorphic implementation of voice activity detection](https://mlsanity.com/paper/1910.09993/) (2019)

## Built on by (most cited first, 30 of 44)

- [Sparse Spiking Gradient Descent](https://mlsanity.com/paper/2105.08810/) (2021)
- [Optimized Potential Initialization for Low-latency Spiking Neural Networks](https://mlsanity.com/paper/2202.01440/) (2022)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding](https://mlsanity.com/paper/2404.01174/) (2024)
- [ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks](https://mlsanity.com/paper/2306.03693/) (2023)
- [A Unified Framework for Soft Threshold Pruning](https://mlsanity.com/paper/2302.13019/) (2023)
- [Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning](https://mlsanity.com/paper/2406.01072/) (2024)
- [Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN](https://mlsanity.com/paper/2403.03409/) (2024)
- [Recent Advances and New Frontiers in Spiking Neural Networks](https://mlsanity.com/paper/2204.07050/) (2022)
- [Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2407.08130/) (2024)
- [Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks](https://mlsanity.com/paper/2211.12219/) (2022)
- [LoAS: Fully Temporal-Parallel Dataflow for Dual-Sparse Spiking Neural Networks](https://mlsanity.com/paper/2407.14073/) (2024)
- [Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks](https://mlsanity.com/paper/2211.12714/) (2022)
- [Sparse Compressed Spiking Neural Network Accelerator for Object Detection](https://mlsanity.com/paper/2205.00778/) (2022)
- [Spiking Variational Graph Representation Inference for Video Summarization](https://mlsanity.com/paper/2508.15389/) (2025)
- [Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network](https://mlsanity.com/paper/2305.12148/) (2023)
- [Resource Constrained Model Compression via Minimax Optimization for Spiking Neural Networks](https://mlsanity.com/paper/2308.04672/) (2023)
- [Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference](https://mlsanity.com/paper/2311.07625/) (2023)
- [Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology](https://mlsanity.com/paper/2312.01213/) (2023)
- [FireFly-S: Exploiting Dual-Side Sparsity for Spiking Neural Networks Acceleration with Reconfigurable Spatial Architecture](https://mlsanity.com/paper/2408.15578/) (2024)
- [SIT: A Bionic and Non-Linear Neuron for Spiking Neural Network](https://mlsanity.com/paper/2203.16117/) (2022)
- [Brain-Inspired Efficient Pruning: Exploiting Criticality in Spiking Neural Networks](https://mlsanity.com/paper/2311.16141/) (2023)
- [Exploiting Heterogeneity in Timescales for Sparse Recurrent Spiking Neural Networks for Energy-Efficient Edge Computing](https://mlsanity.com/paper/2407.06452/) (2024)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Spatio-Temporal Pruning for Compressed Spiking Large Language Models](https://mlsanity.com/paper/2508.20122/) (2025)
- [The Hardware Impact of Quantization and Pruning for Weights in Spiking Neural Networks](https://mlsanity.com/paper/2302.04174/) (2023)
- [Self-building Neural Networks](https://mlsanity.com/paper/2304.01086/) (2023)
- [Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Network](https://mlsanity.com/paper/2309.13302/) (2023)
- [Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models](https://mlsanity.com/paper/2405.00433/) (2024)

## Abstract (arXiv)

Spiking Neural Networks (SNNs) have been attached great importance due to their biological plausibility and high energy-efficiency on neuromorphic chips. As these chips are usually resource-constrained, the compression of SNNs is thus crucial along the road of practical use of SNNs. Most existing methods directly apply pruning approaches in artificial neural networks (ANNs) to SNNs, which ignore the difference between ANNs and SNNs, thus limiting the performance of the pruned SNNs. Besides, these methods are only suitable for shallow SNNs. In this paper, inspired by synaptogenesis and synapse elimination in the neural system, we propose gradient rewiring (Grad R), a joint learning algorithm of connectivity and weight for SNNs, that enables us to seamlessly optimize network structure without retraining. Our key innovation is to redefine the gradient to a new synaptic parameter, allowing better exploration of network structures by taking full advantage of the competition between pruning and regrowth of connections. The experimental results show that the proposed method achieves minimal loss of SNNs' performance on MNIST and CIFAR-10 dataset so far. Moreover, it reaches a ~3.5% accuracy loss under unprecedented 0.73% connectivity, which reveals remarkable structure refining capability in SNNs. Our work suggests that there exists extremely high redundancy in deep SNNs. Our codes are available at https://github.com/Yanqi-Chen/Gradient-Rewiring.