mlsanityData to 7 Oct 2026

Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression

arXiv 2107.12445 ↗First posted 16 Jul 2021cs.NE cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep spiking neural networks (SNNs) have emerged as a potential alternative to traditional deep learning frameworks, due to their promise to provide increased compute efficiency on event-driven neuromorphic hardware. However, to perform well on complex vision applications, most SNN training frameworks yield large inference latency which translates to increased spike activity and reduced energy efficiency. Hence,minimizing average spike activity while preserving accuracy indeep SNNs remains a significant challenge and opportunity.This paper presents a non-iterative SNN training technique thatachieves ultra-high compression with reduced spiking activitywhile maintaining high inference accuracy. In particular, our framework first uses the attention-maps of an un compressed meta-model to yield compressed ANNs. This step can be tuned to support both irregular and structured channel pruning to leverage computational benefits over a broad range of platforms. The framework then performs sparse-learning-based supervised SNN training using direct inputs. During the training, it jointly optimizes the SNN weight, threshold, and leak parameters to drastically minimize the number of time steps required while retaining compression. To evaluate the merits of our approach, we performed experiments with variants of VGG and ResNet, on both CIFAR-10 and CIFAR-100, and VGG16 on Tiny-ImageNet.The SNN models generated through the proposed technique yield SOTA compression ratios of up to 33.4x with no significant drops in accuracy compared to baseline unpruned counterparts. Compared to existing SNN pruning methods, we achieve up to 8.3x higher compression with improved accuracy.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

33 of 55 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  9. 2018Rethinking the Value of Network Pruningcited by 595
  10. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  11. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  12. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 236
21 more
  1. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 222
  2. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  3. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 203
  4. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  5. 2018Long short-term memory and learning-to-learn in networks of spiking neuronscited by 179
  6. 2017SuperSpike: Supervised learning in multi-layer spiking neural networkscited by 178
  7. 2018Understanding Batch Normalizationcited by 157
  8. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 142
  9. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  10. 2019Enabling Spike-based Backpropagation for Training Deep Neural Network Architecturescited by 119
  11. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
  12. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  13. 2020DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networkscited by 70
  14. 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
  15. 2017STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognitioncited by 41
  16. 2016Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardwarecited by 39
  17. 2019Comprehensive SNN Compression Using ADMM Optimization and Activity Regularizationcited by 38
  18. 2020A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNscited by 37
  19. 2019Optimizing the energy consumption of spiking neural networks for neuromorphic applicationscited by 34
  20. 2020Exploring the Connection Between Binary and Spiking Neural Networkscited by 33
  21. 2019ReStoCNet: Residual Stochastic Binary Convolutional Spiking Neural Network for Memory-Efficient Neuromorphic Computingcited by 12

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 2.

7 papers.

  1. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 61
  2. 2021Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?cited by 9
  3. 2022In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Visioncited by 6
  4. 2022Hoyer regularizer is all you need for ultra low-latency spiking neural networkscited by 4
  5. 2023Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technologycited by 3
  6. 2023Razor SNN: Efficient Spiking Neural Network with Temporal Embeddingscited by 2
  7. 2022S3NN: Time Step Reduction of Spiking Surrogate Gradients for Training Energy Efficient Single-Step Spiking Neural Networkscited by 1