# Optimizing Performance of Recurrent Neural Networks on GPUs

arXiv: https://arxiv.org/abs/1604.01946 · HTML: https://mlsanity.com/paper/1604.01946/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2016.

## Built on by (most cited first, 12 of 12)

- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [TorchMD-Net 2.0: Fast Neural Network Potentials for Molecular Simulations](https://mlsanity.com/paper/2402.17660/) (2024)
- [DA-LSTM: A Dynamic Drift-Adaptive Learning Framework for Interval Load Forecasting with LSTM Networks](https://mlsanity.com/paper/2305.08767/) (2023)
- [Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models](https://mlsanity.com/paper/2102.02344/) (2021)
- [A Comparative Study on Neural Architectures and Training Methods for Japanese Speech Recognition](https://mlsanity.com/paper/2106.05111/) (2021)
- [A Distributed Deep Reinforcement Learning Technique for Application Placement in Edge and Fog Computing Environments](https://mlsanity.com/paper/2110.12415/) (2021)
- [Privacy-preserving Federated Learning for Residential Short Term Load Forecasting](https://mlsanity.com/paper/2111.09248/) (2021)
- [Simple Recurrence Improves Masked Language Models](https://mlsanity.com/paper/2205.11588/) (2022)
- [Demystifying BERT: Implications for Accelerator Design](https://mlsanity.com/paper/2104.08335/) (2021)
- [A comprehensive GeoAI review: Progress, Challenges and Outlooks](https://mlsanity.com/paper/2412.11643/) (2024)

## Abstract (arXiv)

As recurrent neural networks become larger and deeper, training times for single networks are rising into weeks or even months. As such there is a significant incentive to improve the performance and scalability of these networks. While GPUs have become the hardware of choice for training and deploying recurrent models, the implementations employed often make use of only basic optimizations for these architectures. In this article we demonstrate that by exposing parallelism between operations within the network, an order of magnitude speedup across a range of network sizes can be achieved over a naive implementation. We describe three stages of optimization that have been incorporated into the fifth release of NVIDIA's cuDNN: firstly optimizing a single cell, secondly a single layer, and thirdly the entire network.