# Accelerating Transformer Decoding via a Hybrid of Self-attention and Recurrent Neural Network

arXiv: https://arxiv.org/abs/1909.02279 · HTML: https://mlsanity.com/paper/1909.02279/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 2 of 2)

- [Training Integer-Only Deep Recurrent Neural Networks](https://mlsanity.com/paper/2212.11791/) (2022)
- [iRNN: Integer-only Recurrent Neural Network](https://mlsanity.com/paper/2109.09828/) (2021)

## Abstract (arXiv)

Due to the highly parallelizable architecture, Transformer is faster to train than RNN-based models and popularly used in machine translation tasks. However, at inference time, each output word requires all the hidden states of the previously generated words, which limits the parallelization capability, and makes it much slower than RNN-based ones. In this paper, we systematically analyze the time cost of different components of both the Transformer and RNN-based model. Based on it, we propose a hybrid network of self-attention and RNN structures, in which, the highly parallelizable self-attention is utilized as the encoder, and the simpler RNN structure is used as the decoder. Our hybrid network can decode 4-times faster than the Transformer. In addition, with the help of knowledge distillation, our hybrid network achieves comparable translation quality to the original Transformer.