mlsanityData to 7 Oct 2026

Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network

arXiv 2407.01645 ↗First posted 1 Jul 2024cs.NE cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Spiking neural network (SNN) is studied in multidisciplinary domains to (i) enable order-of-magnitudes energy-efficient AI inference and (ii) computationally simulate neuro-scientific mechanisms. The lack of discrete theory obstructs the practical application of SNN by limiting its performance and nonlinearity support. We present a new optimization-theoretic perspective of the discrete dynamics of spiking neurons. We prove that a discrete dynamical system of simple integrate-and-fire models approximates the sub-gradient method over unconstrained optimization problems. We practically extend our theory to introduce a novel sign gradient descent (signGD)-based neuronal dynamics that can (i) approximate diverse nonlinearities beyond ReLU and (ii) advance ANN-to-SNN conversion performance in low time steps. Experiments on large-scale datasets show that our technique achieves (i) state-of-the-art performance in ANN-to-SNN conversion and (ii) is the first to convert new DNN architectures, e.g., ConvNext, MLP-Mixer, and ResMLP. We publicly share our source code at https://github.com/snuhcs/snn_signgd .

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 95 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

42 of 95 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2016Layer Normalizationcited by 5,579
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2020Designing Network Design Spacescited by 954
  12. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
30 more
  1. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  2. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  3. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  4. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  5. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  6. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  7. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 236
  8. 2019Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detectioncited by 174
  9. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  10. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
  11. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  12. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  13. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 113
  14. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 111
  15. 2022Attention Spiking Neural Networkscited by 103
  16. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  17. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  18. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 61
  19. 2020T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Codingcited by 45
  20. 2023Reducing ANN-SNN Conversion Error through Residual Membrane Potentialcited by 39
  21. 2020Brain-Inspired Learning on Neuromorphic Substratescited by 35
  22. 2016Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networkscited by 31
  23. 2022An Unsupervised STDP-based Spiking Neural Network Inspired By Biologically Plausible Learning Rules and Connectionscited by 13
  24. 2021BSNN: Towards Faster and Better Conversion of Artificial Neural Networks to Spiking Neural Networks with Bistable Neuronscited by 12
  25. 2022An Exact Mapping From ReLU Networks to Spiking Neural Networkscited by 12
  26. 2022Linear Leaky-Integrate-and-Fire Neuron Model Based Spiking Neural Networks and Its Mapping Relationship to Deep Neural Networkscited by 8
  27. 2019Real-Time Cortical Simulation on Neuromorphic Hardwarecited by 5
  28. 2019Bifurcation Spiking Neural Networkcited by 4
  29. 2022Captivate! Contextual Language Guidance for Parent-Child Interactioncited by 3
  30. 2022Spiking Approximations of the MaxPooling Operation in Deep SNNscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Differential Coding for Training-Free ANN-to-SNN Conversioncited by 9
  2. 2026SpikeVLA: Vision-Language-Action Models with Spiking Neural Networkscited by 1
  3. 2025Integer Binary-Range Alignment Neuron for Spiking Neural Networkscited by 0
  4. 2025SAFA-SNN: Sparsity-Aware On-Device Few-Shot Class-Incremental Learning with Fast-Adaptive Structure of Spiking Neural Networkcited by 0
  5. 2026Error Amplification Limits ANN-to-SNN Conversion in Continuous Controlcited by 0