# Spartus: A 9.4 TOp/s FPGA-based LSTM Accelerator Exploiting Spatio-Temporal Sparsity

arXiv: https://arxiv.org/abs/2108.02297 · HTML: https://mlsanity.com/paper/2108.02297/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA](https://mlsanity.com/paper/1612.00694/) (2016)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps](https://mlsanity.com/paper/1706.01406/) (2017)
- [Alternating Multi-bit Quantization for Recurrent Neural Networks](https://mlsanity.com/paper/1802.00150/) (2018)
- [C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs](https://mlsanity.com/paper/1803.06305/) (2018)
- [EdgeDRNN: Recurrent Neural Network Accelerator for Edge Inference](https://mlsanity.com/paper/2012.13600/) (2020)
- [Delta Networks for Optimized Recurrent Network Computation](https://mlsanity.com/paper/1612.05571/) (2016)
- [Learning Intrinsic Sparse Structures within Long Short-Term Memory](https://mlsanity.com/paper/1709.05027/) (2017)
- [E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs](https://mlsanity.com/paper/1812.07106/) (2018)
- [Recurrent Neural Network Control of a Hybrid Dynamic Transfemoral Prosthesis with EdgeDRNN Accelerator](https://mlsanity.com/paper/2002.03197/) (2020)

## Built on by (most cited first, 20 of 20)

- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [3ET: Efficient Event-based Eye Tracking using a Change-Based ConvLSTM Network](https://mlsanity.com/paper/2308.11771/) (2023)
- [Event-Based Eye Tracking. AIS 2024 Challenge Survey](https://mlsanity.com/paper/2404.11770/) (2024)
- [Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks](https://mlsanity.com/paper/2112.13896/) (2021)
- [To Spike or Not To Spike: A Digital Hardware Perspective on Deep Learning Acceleration](https://mlsanity.com/paper/2306.15749/) (2023)
- [Language Modeling on a SpiNNaker 2 Neuromorphic Chip](https://mlsanity.com/paper/2312.09084/) (2023)
- [Epilepsy Seizure Detection and Prediction using an Approximate Spiking Convolutional Transformer](https://mlsanity.com/paper/2402.09424/) (2024)
- [Skydiver: A Spiking Neural Network Accelerator Exploiting Spatio-Temporal Workload Balance](https://mlsanity.com/paper/2203.07516/) (2022)
- [Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancement](https://mlsanity.com/paper/2408.12425/) (2024)
- [Event-Based Eye Tracking. 2025 Event-based Vision Workshop](https://mlsanity.com/paper/2504.18249/) (2025)
- [DeltaLLM: Delta-Key Sparse Attention for Edge LLMs](https://mlsanity.com/paper/2507.19608/) (2025)
- [Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement](https://mlsanity.com/paper/2411.02019/) (2024)
- [MP-DPD: Low-Complexity Mixed-Precision Neural Networks for Energy-Efficient Digital Predistortion of Wideband Power Amplifiers](https://mlsanity.com/paper/2404.15364/) (2024)
- [Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference](https://mlsanity.com/paper/2311.07625/) (2023)
- [Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training](https://mlsanity.com/paper/2312.09391/) (2023)
- [Quantized Approximately Orthogonal Recurrent Neural Networks](https://mlsanity.com/paper/2402.04012/) (2024)
- [Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models](https://mlsanity.com/paper/2405.00433/) (2024)
- [DPD-NeuralEngine: A 22-nm 6.6-TOPS/W/mm2 Recurrent Neural Network Accelerator for Wideband Power Amplifier Digital Pre-Distortion](https://mlsanity.com/paper/2410.11766/) (2024)
- [Leveraging Recurrent Neural Networks for Predicting Motor Movements from Primate Motor Cortex Neural Recordings](https://mlsanity.com/paper/2410.22283/) (2024)
- [AS-ASR: A Lightweight Framework for Aphasia-Specific Automatic Speech Recognition](https://mlsanity.com/paper/2506.06566/) (2025)

## Abstract (arXiv)

Long Short-Term Memory (LSTM) recurrent networks are frequently used for tasks involving time-sequential data such as speech recognition. Unlike previous LSTM accelerators that either exploit spatial weight sparsity or temporal activation sparsity, this paper proposes a new accelerator called "Spartus" that exploits spatio-temporal sparsity to achieve ultra-low latency inference. Spatial sparsity is induced using a new Column-Balanced Targeted Dropout (CBTD) structured pruning method, producing structured sparse weight matrices for a balanced workload. The pruned networks running on Spartus hardware achieve weight sparsity levels of up to 96% and 94% with negligible accuracy loss on the TIMIT and the Librispeech datasets. To induce temporal sparsity in LSTM, we extend the previous DeltaGRU method to the DeltaLSTM method. Combining spatio-temporal sparsity with CBTD and DeltaLSTM saves on weight memory access and associated arithmetic operations. The Spartus architecture is scalable and supports real-time online speech recognition when implemented on small and large FPGAs. Spartus per-sample latency for a single DeltaLSTM layer of 1024 neurons averages 1 us. Exploiting spatio-temporal sparsity on our test LSTM network using the TIMIT dataset leads to 46X speedup of Spartus over its theoretical hardware performance to achieve 9.4 TOp/s effective batch-1 throughput and 1.1 TOp/s/W power efficiency.