Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks
Not on the map. Why?
Abstract (arXiv)
Brain-inspired neuromorphic computing with spiking neural networks (SNNs) is a promising energy-efficient computational approach. However, successfully training SNNs in a more biologically plausible and neuromorphic-hardware-friendly way is still challenging. Most recent methods leverage spatial and temporal backpropagation (BP), not adhering to neuromorphic properties. Despite the efforts of some online training methods, tackling spatial credit assignments by alternatives with comparable performance as spatial BP remains a significant problem. In this work, we propose a novel method, online pseudo-zeroth-order (OPZO) training. Our method only requires a single forward propagation with noise injection and direct top-down signals for spatial credit assignment, avoiding spatial BP's problem of symmetric weights and separate phases for layer-by-layer forward-backward propagation. OPZO solves the large variance problem of zeroth-order methods by the pseudo-zeroth-order formulation and momentum feedback connections, while having more guarantees than random feedback. Combining online training, OPZO can pave paths to on-chip SNN training. Experiments on neuromorphic and static datasets with fully connected and convolutional networks demonstrate the effectiveness of OPZO with similar performance compared with spatial BP, as well as estimated low training costs.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 44 of its 79 references that are in the map. First posted on arXiv 17 Jul 2024.
Builds on
44 of 79 references.
References · 44
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
- 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
- 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
- 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
- 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
- 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
- 2023Fine-Tuning Language Models with Just Forward Passescited by 345
- 2015Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccadescited by 314
- 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
- 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
- 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
- 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 259
32 more
- 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
- 2016Training Deep Spiking Neural Networks using Backpropagationcited by 205
- 2014Difference Target Propagationcited by 148
- 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 139
- 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 100
- 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
- 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
- 2019Deep Learning without Weight Transportcited by 87
- 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
- 2022Gradients without Backpropagationcited by 73
- 2022Scaling Forward Gradient With Local Lossescited by 65
- 2015How Important is Weight Symmetry in Backpropagation?cited by 58
- 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
- 2019Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performancecited by 46
- 2023SEENN: Towards Temporal Spiking Early-Exit Neural Networkscited by 41
- 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 40
- 2016Neuromorphic Deep Learning Machinescited by 38
- 2020Surrogate gradients for analog neuromorphic computingcited by 38
- 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
- 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
- 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 34
- 2020Online Spatio-Temporal Learning in Deep Neural Networkscited by 32
- 2020EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementationscited by 31
- 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 28
- 2022Exploring Lottery Ticket Hypothesis in Spiking Neural Networkscited by 26
- 2022Training Spiking Neural Networks with Local Tandem Learningcited by 23
- 2023Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networkscited by 18
- 2020Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networkscited by 15
- 2023One Forward is Enough for Neural Network Training via Likelihood Ratio Methodcited by 11
- 2023Zeroth-order Optimization with Weak Dimension Dependencycited by 11
- 2023SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networkscited by 8
- 2022Low-Variance Forward Gradients using Direct Feedback Alignment and Momentumcited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.