# Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings

arXiv: https://arxiv.org/abs/2306.17670 · HTML: https://mlsanity.com/paper/2306.17670/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2003.01811/) (2020)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks](https://mlsanity.com/paper/2303.04347/) (2023)
- [Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks](https://mlsanity.com/paper/2103.12593/) (2021)
- [Temporal-wise Attention Spiking Neural Networks for Event Streams Classification](https://mlsanity.com/paper/2107.11711/) (2021)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Dilated convolution with learnable spacings](https://mlsanity.com/paper/2112.03740/) (2021)
- [STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks](https://mlsanity.com/paper/2210.05241/) (2022)
- [Loss shaping enhances exact gradient learning with Eventprop in spiking neural networks](https://mlsanity.com/paper/2212.01232/) (2022)
- [Axonal Delay As a Short-Term Memory for Feed Forward Deep Spiking Neural Networks](https://mlsanity.com/paper/2205.02115/) (2022)
- [Adaptive Axonal Delays in feedforward spiking neural networks for accurate spoken word recognition](https://mlsanity.com/paper/2302.08607/) (2023)
- [Beyond Weights: Deep learning in Spiking Neural Networks with pure synaptic-delay training](https://mlsanity.com/paper/2306.06237/) (2023)
- [Memory via Temporal Delays in weightless Spiking Neural Network](https://mlsanity.com/paper/2202.07132/) (2022)
- [Dilated Convolution with Learnable Spacings: beyond bilinear interpolation](https://mlsanity.com/paper/2306.00817/) (2023)
- [MSAT: Biologically Inspired Multi-Stage Adaptive Threshold for Conversion of Spiking Neural Networks](https://mlsanity.com/paper/2303.13080/) (2023)

## Built on by (most cited first, 30 of 50)

- [Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models](https://mlsanity.com/paper/2404.18508/) (2024)
- [S7: Selective and Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2410.03464/) (2024)
- [Co-learning synaptic delays, weights and adaptation in spiking neural networks](https://mlsanity.com/paper/2311.16112/) (2023)
- [Balanced Resonate-and-Fire Neurons](https://mlsanity.com/paper/2402.14603/) (2024)
- [DelGrad: Exact event-based gradients for training delays and weights on spiking neuromorphic hardware](https://mlsanity.com/paper/2404.19165/) (2024)
- [A Fully-Configurable Open-Source Software-Defined Digital Quantized Spiking Neural Core Architecture](https://mlsanity.com/paper/2404.02248/) (2024)
- [Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networks](https://mlsanity.com/paper/2411.04760/) (2024)
- [Compute-in-Memory Implementation of State Space Models for Event Sequence Processing](https://mlsanity.com/paper/2511.13912/) (2025)
- [Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate](https://mlsanity.com/paper/2310.14982/) (2023)
- [Adapting to time: Why nature may have evolved a diverse set of neurons](https://mlsanity.com/paper/2404.14325/) (2024)
- [Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning](https://mlsanity.com/paper/2405.16851/) (2024)
- [The Role of Temporal Hierarchy in Spiking Neural Networks](https://mlsanity.com/paper/2407.18838/) (2024)
- [Scaling Up Resonate-and-Fire Networks for Fast Deep Learning](https://mlsanity.com/paper/2504.00719/) (2025)
- [State-Space Model Inspired Multiple-Input Multiple-Output Spiking Neurons](https://mlsanity.com/paper/2504.02591/) (2025)
- [SiLIF: Structured State Space Model Dynamics and Parametrization for Spiking Neural Networks](https://mlsanity.com/paper/2506.06374/) (2025)
- [Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networks](https://mlsanity.com/paper/2507.16043/) (2025)
- [EchoSpike Predictive Plasticity: An Online Local Learning Rule for Spiking Neural Networks](https://mlsanity.com/paper/2405.13976/) (2024)
- [STREAM: A Universal State-Space Model for Sparse Geometric Data](https://mlsanity.com/paper/2411.12603/) (2024)
- [Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness](https://mlsanity.com/paper/2503.17394/) (2025)
- [Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation](https://mlsanity.com/paper/2506.23717/) (2025)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [A Complete Pipeline for deploying SNNs with Synaptic Delays on Loihi 2](https://mlsanity.com/paper/2510.13757/) (2025)
- [Exploiting heterogeneous delays for efficient computation in low-bit neural networks](https://mlsanity.com/paper/2510.27434/) (2025)
- [SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognition](https://mlsanity.com/paper/2511.07883/) (2025)
- [Hardware-accelerated graph neural networks: an alternative approach for neuromorphic event-based audio classification and keyword spotting on SoC FPGA](https://mlsanity.com/paper/2602.16442/) (2026)
- [Bullet Trains: Parallelizing Training of Temporally Precise Spiking Neural Networks](https://mlsanity.com/paper/2603.13283/) (2026)
- [SpikingJET: Enhancing Fault Injection for Fully and Convolutional Spiking Neural Networks](https://mlsanity.com/paper/2404.00383/) (2024)
- [Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks](https://mlsanity.com/paper/2404.14024/) (2024)
- [Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models](https://mlsanity.com/paper/2405.00433/) (2024)
- [Decoding finger velocity from cortical spike trains with recurrent spiking neural networks](https://mlsanity.com/paper/2409.01762/) (2024)

## Abstract (arXiv)

Spiking Neural Networks (SNNs) are a promising research direction for building power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it is well-known that spiking neurons respond more strongly to coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms to learn these delays have been lacking. Here, we propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between consecutive layers, we use 1D convolutions across time. The kernels contain only a few non-zero weights - one per synapse - whose positions correspond to the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS). We evaluated our method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its non-spiking version Google Speech Commands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outperformed the state-of-the-art in the three datasets without using recurrent connections and with substantially fewer parameters. Our work demonstrates the potential of delay learning in developing accurate and precise models for temporal data processing. Our code is based on PyTorch / SpikingJelly and available at: https://github.com/Thvnvtos/SNN-delays