# Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware

arXiv: https://arxiv.org/abs/1601.04187 · HTML: https://mlsanity.com/paper/1601.04187/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2016.

## Built on by (most cited first, 30 of 40)

- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration](https://mlsanity.com/paper/2106.06984/) (2021)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks](https://mlsanity.com/paper/2210.13768/) (2022)
- [Neuromorphic Data Augmentation for Training Spiking Neural Networks](https://mlsanity.com/paper/2203.06145/) (2022)
- [Sparse Spiking Gradient Descent](https://mlsanity.com/paper/2105.08810/) (2021)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/2104.10719/) (2021)
- [Federated Learning with Spiking Neural Networks](https://mlsanity.com/paper/2106.06579/) (2021)
- [High-performance deep spiking neural networks with 0.3 spikes per neuron](https://mlsanity.com/paper/2306.08744/) (2023)
- [Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence](https://mlsanity.com/paper/2106.01288/) (2021)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Microsaccade-inspired Event Camera for Robotics](https://mlsanity.com/paper/2405.17769/) (2024)
- [An Exact Mapping From ReLU Networks to Spiking Neural Networks](https://mlsanity.com/paper/2212.12522/) (2022)
- [Spiking Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/2406.19230/) (2024)
- [Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?](https://mlsanity.com/paper/2112.12133/) (2021)
- [Linear Leaky-Integrate-and-Fire Neuron Model Based Spiking Neural Networks and Its Mapping Relationship to Deep Neural Networks](https://mlsanity.com/paper/2207.04889/) (2022)
- [Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression](https://mlsanity.com/paper/2107.12445/) (2021)
- [Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration](https://mlsanity.com/paper/2205.10121/) (2022)
- [SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks](https://mlsanity.com/paper/2206.09449/) (2022)
- [In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision](https://mlsanity.com/paper/2212.10881/) (2022)
- [SparseProp: Efficient Event-Based Simulation and Training of Sparse Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2312.17216/) (2023)
- [SMS: Spiking Marching Scheme for Efficient Long Time Integration of Differential Equations](https://mlsanity.com/paper/2211.09928/) (2022)
- [FTBC: Forward Temporal Bias Correction for Optimizing ANN-SNN Conversion](https://mlsanity.com/paper/2403.18388/) (2024)
- [Spiking Hyperdimensional Network: Neuromorphic Models Integrated with Memory-Inspired Framework](https://mlsanity.com/paper/2110.00214/) (2021)
- [Hoyer regularizer is all you need for ultra low-latency spiking neural networks](https://mlsanity.com/paper/2212.10170/) (2022)
- [SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network](https://mlsanity.com/paper/2310.06488/) (2023)
- [HYPER-SNN: Towards Energy-efficient Quantized Deep Spiking Neural Networks for Hyperspectral Image Classification](https://mlsanity.com/paper/2107.11979/) (2021)

## Abstract (arXiv)

In recent years the field of neuromorphic low-power systems that consume orders of magnitude less power gained significant momentum. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, efficient processing of temporal sequences or variable length-inputs remain difficult. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This "train-and-constrain" method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find that short synaptic delays are sufficient to implement the dynamical (temporal) aspect of the RNN in the question classification task. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ~17 uW.