# Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies

arXiv: https://arxiv.org/abs/2304.12760 · HTML: https://mlsanity.com/paper/2304.12760/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 68 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2003.01811/) (2020)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [Parallelizing Linear Recurrent Neural Nets Over Sequence Length](https://mlsanity.com/paper/1709.04057/) (2017)
- [Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks](https://mlsanity.com/paper/2303.04347/) (2023)
- [Attention Spiking Neural Networks](https://mlsanity.com/paper/2209.13929/) (2022)
- [Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation](https://mlsanity.com/paper/2205.00459/) (2022)
- [Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch](https://mlsanity.com/paper/2010.01729/) (2020)
- [Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks](https://mlsanity.com/paper/2105.11654/) (2021)

## Built on by (most cited first, 30 of 41)

- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models](https://mlsanity.com/paper/2404.18508/) (2024)
- [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://mlsanity.com/paper/2406.03287/) (2024)
- [SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models](https://mlsanity.com/paper/2408.14909/) (2024)
- [Rethinking Spiking Neural Networks from an Ensemble Learning Perspective](https://mlsanity.com/paper/2502.14218/) (2025)
- [LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Model](https://mlsanity.com/paper/2402.00411/) (2024)
- [SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learning](https://mlsanity.com/paper/2410.17268/) (2024)
- [EZSR: Event-based Zero-Shot Recognition](https://mlsanity.com/paper/2407.21616/) (2024)
- [Brain-Inspired Online Adaptation for Remote Sensing with Spiking Neural Network](https://mlsanity.com/paper/2409.02146/) (2024)
- [Efficient Parallel Training Methods for Spiking Neural Networks with Constant Time Complexity](https://mlsanity.com/paper/2506.12087/) (2025)
- [Time Is All It Takes: Spike-Retiming Attacks on Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2602.03284/) (2026)
- [Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time](https://mlsanity.com/paper/2505.18023/) (2025)
- [SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network](https://mlsanity.com/paper/2408.00788/) (2024)
- [Faster and Stronger: When ANN-SNN Conversion Meets Parallel Spiking Calculation](https://mlsanity.com/paper/2412.13610/) (2024)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling](https://mlsanity.com/paper/2509.17186/) (2025)
- [Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing](https://mlsanity.com/paper/2601.00245/) (2026)
- [Bullet Trains: Parallelizing Training of Temporally Precise Spiking Neural Networks](https://mlsanity.com/paper/2603.13283/) (2026)
- [Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models](https://mlsanity.com/paper/2405.00433/) (2024)
- [SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation](https://mlsanity.com/paper/2409.04082/) (2024)
- [Time-independent Spiking Neuron via Membrane Potential Estimation for Efficient Spiking Neural Networks](https://mlsanity.com/paper/2409.04978/) (2024)
- [Efficient Training of Spiking Neural Networks by Spike-aware Data Pruning](https://mlsanity.com/paper/2510.04098/) (2025)
- [Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients](https://mlsanity.com/paper/2605.27412/) (2026)
- [NiSNN-A: Non-iterative Spiking Neural Networks with Attention with Application to Motor Imagery EEG Classification](https://mlsanity.com/paper/2312.05643/) (2023)
- [Topology Optimization of Random Memristors for Input-Aware Dynamic SNN](https://mlsanity.com/paper/2407.18625/) (2024)
- [Obtaining Optimal Spiking Neural Network in Sequence Learning via CRNN-SNN Conversion](https://mlsanity.com/paper/2408.09403/) (2024)
- [Network Representation Learning for Biophysical Neural Network Analysis](https://mlsanity.com/paper/2410.11503/) (2024)
- [DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks](https://mlsanity.com/paper/2503.10052/) (2025)
- [Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNN](https://mlsanity.com/paper/2504.17751/) (2025)
- [A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2506.14464/) (2025)

## Abstract (arXiv)

Vanilla spiking neurons in Spiking Neural Networks (SNNs) use charge-fire-reset neuronal dynamics, which can only be simulated serially and can hardly learn long-time dependencies. We find that when removing reset, the neuronal dynamics can be reformulated in a non-iterative form and parallelized. By rewriting neuronal dynamics without reset to a general formulation, we propose the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. To avoid the use of future inputs for step-by-step inference, the weights of the PSN can be masked, resulting in the masked PSN. By sharing weights across time-steps based on the masked PSN, the sliding PSN is proposed to handle sequences of varying lengths. We evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family in efficiency and accuracy. To the best of our knowledge, this is the first study about parallelizing spiking neurons and can be a cornerstone for the spiking deep learning research. Our codes are available at urlhttps://github.com/fangwei123456/Parallel-Spiking-Neuron.