# A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware

arXiv: https://arxiv.org/abs/2107.03992 · HTML: https://mlsanity.com/paper/2107.03992/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 8 Jul 2021.

## Builds on (most cited first, 9 of 9)

- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Deep Rewiring: Training very sparse deep networks](https://mlsanity.com/paper/1711.05136/) (2017)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [Long Short-Term Memory Spiking Networks and Their Applications](https://mlsanity.com/paper/2007.04779/) (2020)

## Built on by (most cited first, 30 of 32)

- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Inherent Redundancy in Spiking Neural Networks](https://mlsanity.com/paper/2308.08227/) (2023)
- [Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning](https://mlsanity.com/paper/2401.05444/) (2024)
- [Spiking neural networks for nonlinear regression](https://mlsanity.com/paper/2210.03515/) (2022)
- [Spiking Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/2406.19230/) (2024)
- [DelGrad: Exact event-based gradients for training delays and weights on spiking neuromorphic hardware](https://mlsanity.com/paper/2404.19165/) (2024)
- [Classical and Quantum Physical Reservoir Computing for Onboard Artificial Intelligence Systems: A Perspective](https://mlsanity.com/paper/2407.04717/) (2024)
- [Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks](https://mlsanity.com/paper/2402.11984/) (2024)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [Reservoir Computing Using Measurement-Controlled Quantum Dynamics](https://mlsanity.com/paper/2403.01024/) (2024)
- [Language Modeling on a SpiNNaker 2 Neuromorphic Chip](https://mlsanity.com/paper/2312.09084/) (2023)
- [RSC-SNN: Exploring the Trade-off Between Adversarial Robustness and Accuracy in Spiking Neural Networks via Randomized Smoothing Coding](https://mlsanity.com/paper/2407.20099/) (2024)
- [A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing](https://mlsanity.com/paper/2409.15022/) (2024)
- [The Physics of Preference: Unravelling Imprecision of Human Preferences through Magnetisation Dynamics](https://mlsanity.com/paper/2310.00267/) (2023)
- [Physical Reservoir Computing Enabled by Solitary Waves and Biologically-Inspired Nonlinear Transformation of Input Data](https://mlsanity.com/paper/2402.03319/) (2024)
- [NeuroVLA: A Brain-inspired Embodied Intelligence for Fluid and Fast Reflexive Robotics Control](https://mlsanity.com/paper/2601.14628/) (2026)
- [SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network](https://mlsanity.com/paper/2310.06488/) (2023)
- [Beyond representational alignment with brain-guided language models for robust reasoning](https://mlsanity.com/paper/2606.11893/) (2026)
- [Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network](https://mlsanity.com/paper/2305.12148/) (2023)
- [GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL](https://mlsanity.com/paper/2404.15597/) (2024)
- [A differentiable brain simulator bridging brain simulation and brain-inspired computing](https://mlsanity.com/paper/2311.05106/) (2023)
- [FeNN: A RISC-V vector processor for Spiking Neural Network acceleration](https://mlsanity.com/paper/2506.11760/) (2025)
- [Memory-enriched computation and learning in spiking neural networks through Hebbian plasticity](https://mlsanity.com/paper/2205.11276/) (2022)
- [Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement](https://mlsanity.com/paper/2401.10310/) (2024)
- [IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing](https://mlsanity.com/paper/2507.07396/) (2025)
- [Low Power Neuromorphic EMG Gesture Classification](https://mlsanity.com/paper/2206.02061/) (2022)

## Abstract (arXiv)

Spike-based neuromorphic hardware holds the promise to provide more energy efficient implementations of Deep Neural Networks (DNNs) than standard hardware such as GPUs. But this requires to understand how DNNs can be emulated in an event-based sparse firing regime, since otherwise the energy-advantage gets lost. In particular, DNNs that solve sequence processing tasks typically employ Long Short-Term Memory (LSTM) units that are hard to emulate with few spikes. We show that a facet of many biological neurons, slow after-hyperpolarizing (AHP) currents after each spike, provides an efficient solution. AHP-currents can easily be implemented in neuromorphic hardware that supports multi-compartment neuron models, such as Intel's Loihi chip. Filter approximation theory explains why AHP-neurons can emulate the function of LSTM units. This yields a highly energy-efficient approach to time series classification. Furthermore it provides the basis for implementing with very sparse firing an important class of large DNNs that extract relations between words and sentences in a text in order to answer questions about the text.