mlsanityData to 7 Oct 2026

SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks

arXiv 2404.19668 ↗First posted 15 Apr 2024cs.NE cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Weight quantization is used to deploy high-performance deep learning models on resource-limited hardware, enabling the use of low-precision integers for storage and computation. Spiking neural networks (SNNs) share the goal of enhancing efficiency, but adopt an 'event-driven' approach to reduce the power consumption of neural network inference. While extensive research has focused on weight quantization, quantization-aware training (QAT), and their application to SNNs, the precision reduction of state variables during training has been largely overlooked, potentially diminishing inference performance. This paper introduces two QAT schemes for stateful neurons: (i) a uniform quantization strategy, an established method for weight quantization, and (ii) threshold-centered quantization, which allocates exponentially more quantization levels near the firing threshold. Our results show that increasing the density of quantization levels around the firing threshold improves accuracy across several benchmark datasets. We provide an ablation analysis of the effects of weight and state quantization, both individually and combined, and how they impact models. Our comprehensive empirical evaluation includes full precision, 8-bit, 4-bit, and 2-bit quantized SNNs, using QAT, stateful QAT (SQUAT), and post-training quantization methods. The findings indicate that the combination of QAT and SQUAT enhance performance the most, but given the choice of one or the other, QAT improves performance by the larger degree. These trends are consistent all datasets. Our methods have been made available in our Python library snnTorch: https://github.com/jeshraghian/snntorch.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

37 of 62 references.

References · 37

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  6. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  9. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  10. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  11. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  12. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
25 more
  1. 2019The Heidelberg spiking datasets for the systematic evaluation of spiking neural networkscited by 135
  2. 2019Loss Aware Post-training Quantizationcited by 94
  3. 2021Efficient Neuromorphic Signal Processing with Loihi 2cited by 90
  4. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  5. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  6. 2015Spiking Deep Networks with LIF Neuronscited by 63
  7. 2017Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimizationcited by 56
  8. 2019Resource Efficient 3D Convolutional Neural Networkscited by 50
  9. 2017STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognitioncited by 41
  10. 2016Neuromorphic Deep Learning Machinescited by 38
  11. 2020Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applicationscited by 34
  12. 2021Q-SpiNN: A Framework for Quantizing Spiking Neural Networkscited by 33
  13. 2016Ternary Neural Networks for Resource-Efficient AI Applicationscited by 30
  14. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
  15. 2018Adversarial Noise Layer: Regularize Neural Network By Adding Noisecited by 25
  16. 2023Neuromorphic Intermediate Representation: A Unified Instruction Set for Interoperable Brain-Inspired Computingcited by 22
  17. 2017Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networkscited by 21
  18. 2022Spiking Neural Networks for Frame-based and Event-based Single Object Localizationcited by 18
  19. 2021Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networkscited by 16
  20. 2020Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networkscited by 15
  21. 2022Sub-mW Neuromorphic SNN audio processing applications with Rockpool and Xylocited by 12
  22. 2021In-Hardware Learning of Multilayer Spiking Neural Networks on a Neuromorphic Processorcited by 11
  23. 2022Spiking neural networks for nonlinear regressioncited by 10
  24. 2022Training Discrete Deep Generative Models via Gapped Straight-Through Estimatorcited by 8
  25. 2021Hessian Aware Quantization of Spiking Neural Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Scalable MatMul-free Language Modelingcited by 21
  2. 2025Hyperdimensional Decoding of Spiking Neural Networkscited by 0