# E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs

arXiv: https://arxiv.org/abs/1812.07106 · HTML: https://mlsanity.com/paper/1812.07106/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2018.

## Built on by (most cited first, 8 of 8)

- [Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design](https://mlsanity.com/paper/2209.09570/) (2022)
- [Spartus: A 9.4 TOp/s FPGA-based LSTM Accelerator Exploiting Spatio-Temporal Sparsity](https://mlsanity.com/paper/2108.02297/) (2021)
- [Ditto: Accelerating Diffusion Model via Temporal Value Similarity](https://mlsanity.com/paper/2501.11211/) (2025)
- [GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity](https://mlsanity.com/paper/2108.11033/) (2021)
- [Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers](https://mlsanity.com/paper/2112.15348/) (2021)
- [DPD-NeuralEngine: A 22-nm 6.6-TOPS/W/mm2 Recurrent Neural Network Accelerator for Wideband Power Amplifier Digital Pre-Distortion](https://mlsanity.com/paper/2410.11766/) (2024)
- [MOHAQ: Multi-Objective Hardware-Aware Quantization of Recurrent Neural Networks](https://mlsanity.com/paper/2108.01192/) (2021)
- [Coflex: Enhancing HW-NAS with Sparse Gaussian Processes for Efficient and Scalable DNN Accelerator Design](https://mlsanity.com/paper/2507.23437/) (2025)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) are becoming increasingly important for time series-related applications which require efficient and real-time implementations. The two major types are Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. It is a challenging task to have real-time, efficient, and accurate hardware RNN implementations because of the high sensitivity to imprecision accumulation and the requirement of special activation function implementations. A key limitation of the prior works is the lack of a systematic design optimization framework of RNN model and hardware implementations, especially when the block size (or compression ratio) should be jointly optimized with RNN type, layer size, etc. In this paper, we adopt the block-circulant matrix-based framework, and present the Efficient RNN (E-RNN) framework for FPGA implementations of the Automatic Speech Recognition (ASR) application. The overall goal is to improve performance/energy efficiency under accuracy requirement. We use the alternating direction method of multipliers (ADMM) technique for more accurate block-circulant training, and present two design explorations providing guidance on block size and reducing RNN training trials. Based on the two observations, we decompose E-RNN in two phases: Phase I on determining RNN model to reduce computation and storage subject to accuracy requirement, and Phase II on hardware implementations given RNN model, including processing element design/optimization, quantization, activation implementation, etc. Experimental results on actual FPGA deployments show that E-RNN achieves a maximum energy efficiency improvement of 37.4× compared with ESE, and more than 2× compared with C-LSTM, under the same accuracy.