# Saving RNN Computations with a Neuron-Level Fuzzy Memoization Scheme

arXiv: https://arxiv.org/abs/2202.06563 · HTML: https://mlsanity.com/paper/2202.06563/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 19 of 19)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge](https://mlsanity.com/paper/1609.06647/) (2016)
- [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://mlsanity.com/paper/1602.02830/) (2016)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Massive Exploration of Neural Machine Translation Architectures](https://mlsanity.com/paper/1703.03906/) (2017)
- [ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA](https://mlsanity.com/paper/1612.00694/) (2016)
- [Bitwise Neural Networks](https://mlsanity.com/paper/1601.06071/) (2016)
- [The High-Dimensional Geometry of Binary Neural Networks](https://mlsanity.com/paper/1705.07199/) (2017)
- [E-PUR: An Energy-Efficient Processing Unit for Recurrent Neural Networks](https://mlsanity.com/paper/1711.07480/) (2017)
- [FPGA-Based Low-Power Speech Recognition with Recurrent Neural Networks](https://mlsanity.com/paper/1610.00552/) (2016)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) are a key technology for applications such as automatic speech recognition or machine translation. Unlike conventional feed-forward DNNs, RNNs remember past information to improve the accuracy of future predictions and, therefore, they are very effective for sequence processing problems. For each application run, recurrent layers are executed many times for processing a potentially large sequence of inputs (words, images, audio frames, etc.). In this paper, we observe that the output of a neuron exhibits small changes in consecutive invocations.~We exploit this property to build a neuron-level fuzzy memoization scheme, which dynamically caches each neuron's output and reuses it whenever it is predicted that the current output will be similar to a previously computed result, avoiding in this way the output computations. The main challenge in this scheme is determining whether the new neuron's output for the current input in the sequence will be similar to a recently computed result. To this end, we extend the recurrent layer with a much simpler Bitwise Neural Network (BNN), and show that the BNN and RNN outputs are highly correlated: if two BNN outputs are very similar, the corresponding outputs in the original RNN layer are likely to exhibit negligible changes. The BNN provides a low-cost and effective mechanism for deciding when fuzzy memoization can be applied with a small impact on accuracy. We evaluate our memoization scheme on top of a state-of-the-art accelerator for RNNs, for a variety of different neural networks from multiple application domains. We show that our technique avoids more than 26.7% of computations, resulting in 21% energy savings and 1.4x speedup on average.