# Intelligence Processing Units Accelerate Neuromorphic Learning

arXiv: https://arxiv.org/abs/2211.10725 · HTML: https://mlsanity.com/paper/2211.10725/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 60 references that are in the map.
- First posted on arXiv 19 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Efficient Neuromorphic Signal Processing with Loihi 2](https://mlsanity.com/paper/2111.03746/) (2021)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [Spiking Deep Networks with LIF Neurons](https://mlsanity.com/paper/1510.08829/) (2015)
- [Gradient Descent for Spiking Neural Networks](https://mlsanity.com/paper/1706.04698/) (2017)
- [BindsNET: A machine learning-oriented spiking neural networks library in Python](https://mlsanity.com/paper/1806.01423/) (2018)
- [ReckOn: A 28nm Sub-mm2 Task-Agnostic Spiking Recurrent Neural Network Processor Enabling On-Chip Learning over Second-Long Timescales](https://mlsanity.com/paper/2208.09759/) (2022)
- [Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications](https://mlsanity.com/paper/2007.05657/) (2020)
- [The Backpropagation Algorithm Implemented on Spiking Neuromorphic Hardware](https://mlsanity.com/paper/2106.07030/) (2021)
- [BioGrad: Biologically Plausible Gradient-Based Learning for Spiking Neural Networks](https://mlsanity.com/paper/2110.14092/) (2021)

## Built on by (most cited first, 3 of 3)

- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models](https://mlsanity.com/paper/2311.04386/) (2023)
- [Tricking AI chips into Simulating the Human Brain: A Detailed Performance Analysis](https://mlsanity.com/paper/2301.13637/) (2023)

## Abstract (arXiv)

Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency when performing inference with deep learning workloads. Error backpropagation is presently regarded as the most effective method for training SNNs, but in a twist of irony, when training on modern graphics processing units (GPUs) this becomes more expensive than non-spiking networks. The emergence of Graphcore's Intelligence Processing Units (IPUs) balances the parallelized nature of deep learning workloads with the sequential, reusable, and sparsified nature of operations prevalent when training SNNs. IPUs adopt multi-instruction multi-data (MIMD) parallelism by running individual processing threads on smaller data blocks, which is a natural fit for the sequential, non-vectorized steps required to solve spiking neuron dynamical state equations. We present an IPU-optimized release of our custom SNN Python package, snnTorch, which exploits fine-grained parallelism by utilizing low-level, pre-compiled custom operations to accelerate irregular and sparse data access patterns that are characteristic of training SNN workloads. We provide a rigorous performance assessment across a suite of commonly used spiking neuron models, and propose methods to further reduce training run-time via half-precision training. By amortizing the cost of sequential processing into vectorizable population codes, we ultimately demonstrate the potential for integrating domain-specific accelerators with the next generation of neural networks.