# Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2110.02865 · HTML: https://mlsanity.com/paper/2110.02865/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks](https://mlsanity.com/paper/2103.12593/) (2021)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)
- [Supervised learning based on temporal coding in spiking neural networks](https://mlsanity.com/paper/1606.08165/) (2016)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [Fast and energy-efficient neuromorphic deep learning with first-spike times](https://mlsanity.com/paper/1912.11443/) (2019)
- [Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performance](https://mlsanity.com/paper/1909.10837/) (2019)
- [Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1705.02411/) (2017)
- [Spiking neurons with short-term synaptic plasticity form superior generative networks](https://mlsanity.com/paper/1709.08166/) (2017)
- [Introducing "Neuromorphic Computing and Engineering"](https://mlsanity.com/paper/2106.01329/) (2021)
- [Spike time displacement based error backpropagation in convolutional spiking neural networks](https://mlsanity.com/paper/2108.13621/) (2021)
- [Optimality of short-term synaptic plasticity in modelling certain dynamic environments](https://mlsanity.com/paper/2009.06808/) (2020)

## Built on by (most cited first, 5 of 5)

- [NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems](https://mlsanity.com/paper/2304.04640/) (2023)
- [Low Latency Conversion of Artificial Neural Network Models to Rate-encoded Spiking Neural Networks](https://mlsanity.com/paper/2211.08410/) (2022)
- [Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes](https://mlsanity.com/paper/2306.12045/) (2023)
- [SpikeSTAG: Spatial-Temporal Forecasting via GNN-SNN Collaboration](https://mlsanity.com/paper/2508.02069/) (2025)
- [Input Convex Lipschitz Recurrent Neural Networks for Robust and Efficient Process Modeling and Optimization](https://mlsanity.com/paper/2401.07494/) (2024)

## Abstract (arXiv)

Biological spiking neural networks (SNNs) can temporally encode information in their outputs, e.g. in the rank order in which neurons fire, whereas artificial neural networks (ANNs) conventionally do not. As a result, models of SNNs for neuromorphic computing are regarded as potentially more rapid and efficient than ANNs when dealing with temporal input. On the other hand, ANNs are simpler to train, and usually achieve superior performance. Here we show that temporal coding such as rank coding (RC) inspired by SNNs can also be applied to conventional ANNs such as LSTMs, and leads to computational savings and speedups. In our RC for ANNs, we apply backpropagation through time using the standard real-valued activations, but only from a strategically early time step of each sequential input example, decided by a threshold-crossing event. Learning then incorporates naturally also *when* to produce an output, without other changes to the model or the algorithm. Both the forward and the backward training pass can be significantly shortened by skipping the remaining input sequence after that first event. RC-training also significantly reduces time-to-insight during inference, with a minimal decrease in accuracy. The desired speed-accuracy trade-off is tunable by varying the threshold or a regularization parameter that rewards output entropy. We demonstrate these in two toy problems of sequence classification, and in a temporally-encoded MNIST dataset where our RC model achieves 99.19% accuracy after the first input time-step, outperforming the state of the art in temporal coding with SNNs, as well as in spoken-word classification of Google Speech Commands, outperforming non-RC-trained early inference with LSTMs.