# Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Network

arXiv: https://arxiv.org/abs/2309.13302 · HTML: https://mlsanity.com/paper/2309.13302/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 23 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Generative Low-bitwidth Data Free Quantization](https://mlsanity.com/paper/2003.03603/) (2020)
- [Adaptive Quantization for Deep Neural Network](https://mlsanity.com/paper/1712.01048/) (2017)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition](https://mlsanity.com/paper/1710.04734/) (2017)
- [Pruning of Deep Spiking Neural Networks through Gradient Rewiring](https://mlsanity.com/paper/2105.04916/) (2021)
- [CondenseNet V2: Sparse Feature Reactivation for Deep Networks](https://mlsanity.com/paper/2104.04382/) (2021)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines](https://mlsanity.com/paper/1511.04484/) (2015)
- [Spiking Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.17046/) (2023)
- [Structure-Aware Network for Lane Marker Extraction with Dynamic Vision Sensor](https://mlsanity.com/paper/2008.06204/) (2020)
- [Data Level Lottery Ticket Hypothesis for Vision Transformers](https://mlsanity.com/paper/2211.01484/) (2022)
- [Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network](https://mlsanity.com/paper/2305.12148/) (2023)

## Built on by (most cited first, 1 of 1)

- [Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model](https://mlsanity.com/paper/2402.19150/) (2024)

## Abstract (arXiv)

Deploying energy-efficient deep learning algorithms on computational-limited devices, such as robots, is still a pressing issue for real-world applications. Spiking Neural Networks (SNNs), a novel brain-inspired algorithm, offer a promising solution due to their low-latency and low-energy properties over traditional Artificial Neural Networks (ANNs). Despite their advantages, the dense structure of deep SNNs can still result in extra energy consumption. The Lottery Ticket Hypothesis (LTH) posits that within dense neural networks, there exist winning Lottery Tickets (LTs), namely sub-networks, that can be obtained without compromising performance. Inspired by this, this paper delves into the spiking-based LTs (SLTs), examining their unique properties and potential for extreme efficiency. Then, two significant sparse textbftextitRewards are gained through comprehensive explorations and meticulous experiments on SLTs across various dense structures. Moreover, a sparse algorithm tailored for spiking transformer structure, which incorporates convolution operations into the Patch Embedding Projection (ConvPEP) module, has been proposed to achieve Multi-level Sparsity (MultiSp). MultiSp refers to (1) Patch number sparsity; (2) ConvPEP weights sparsity and binarization; and (3) ConvPEP activation layer binarization. Extensive experiments demonstrate that our method achieves extreme sparsity with only a slight performance decrease, paving the way for deploying energy-efficient neural networks in robotics and beyond.