# Recurrent Neural Networks With Limited Numerical Precision

arXiv: https://arxiv.org/abs/1608.06902 · HTML: https://mlsanity.com/paper/1608.06902/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2016.

## Built on by (most cited first, 12 of 12)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Q-S5: Towards Quantized State Space Models](https://mlsanity.com/paper/2406.09477/) (2024)
- [Training Integer-Only Deep Recurrent Neural Networks](https://mlsanity.com/paper/2212.11791/) (2022)
- [iRNN: Integer-only Recurrent Neural Network](https://mlsanity.com/paper/2109.09828/) (2021)
- [Quantizing Recursive Reasoning Models](https://mlsanity.com/paper/2607.16237/) (2026)
- [Tree-based Text-Vision BERT for Video Search in Baidu Video Advertising](https://mlsanity.com/paper/2209.08759/) (2022)
- [Quantized Approximately Orthogonal Recurrent Neural Networks](https://mlsanity.com/paper/2402.04012/) (2024)
- [BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference](https://mlsanity.com/paper/2501.14495/) (2025)
- [An Estimator for the Sensitivity to Perturbations of Deep Neural Networks](https://mlsanity.com/paper/2307.12679/) (2023)
- [HadamRNN: Binary and Sparse Ternary Orthogonal RNNs](https://mlsanity.com/paper/2502.00047/) (2025)
- [QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models](https://mlsanity.com/paper/2507.06079/) (2025)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high. Therefore, there is a great interest in optimizing the computations performed with these models especially when considering development of specialized low-power hardware for deep networks. One way of reducing the computational needs is to limit the numerical precision of the network weights and biases. This has led to different proposed rounding methods which have been applied so far to only Convolutional Neural Networks and Fully-Connected Networks. This paper addresses the question of how to best reduce weight precision during training in the case of RNNs. We present results from the use of different stochastic and deterministic reduced precision training methods applied to three major RNN types which are then tested on several datasets. The results show that the weight binarization methods do not work with the RNNs. However, the stochastic and deterministic ternarization, and pow2-ternarization methods gave rise to low-precision RNNs that produce similar and even higher accuracy on certain datasets therefore providing a path towards training more efficient implementations of RNNs in specialized hardware.