# Accelerating Neural Transformer via an Average Attention Network

arXiv: https://arxiv.org/abs/1805.00631 · HTML: https://mlsanity.com/paper/1805.00631/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 22 of 22)

- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://mlsanity.com/paper/2403.09636/) (2024)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation](https://mlsanity.com/paper/2106.00169/) (2021)
- [Examining Scaling and Transfer of Language Model Architectures for Machine Translation](https://mlsanity.com/paper/2202.00528/) (2022)
- [QaNER: Prompting Question Answering Models for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2203.01543/) (2022)
- [Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders](https://mlsanity.com/paper/2206.02079/) (2022)
- [WeChat Neural Machine Translation Systems for WMT21](https://mlsanity.com/paper/2108.02401/) (2021)
- [An Efficient Transformer Decoder with Compressed Sub-layers](https://mlsanity.com/paper/2101.00542/) (2021)
- [CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling](https://mlsanity.com/paper/2210.07661/) (2022)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Cross-layer Attention Sharing for Pre-trained Large Language Models](https://mlsanity.com/paper/2408.01890/) (2024)
- [Bag of Tricks for Optimizing Transformer Efficiency](https://mlsanity.com/paper/2109.04030/) (2021)
- [The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation](https://mlsanity.com/paper/2205.06618/) (2022)
- [Discrete Cosine Transform as Universal Sentence Encoder](https://mlsanity.com/paper/2106.00934/) (2021)
- [Distributionally Robust Recurrent Decoders with Random Network Distillation](https://mlsanity.com/paper/2110.13229/) (2021)
- [FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer](https://mlsanity.com/paper/2206.15312/) (2022)
- [Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translation](https://mlsanity.com/paper/2106.10002/) (2021)
- [BJTU-WeChat's Systems for the WMT22 Chat Translation Task](https://mlsanity.com/paper/2211.15009/) (2022)
- [Dual-Pipeline with Low-Rank Adaptation for New Language Integration in Multilingual ASR](https://mlsanity.com/paper/2406.07842/) (2024)
- [Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruning](https://mlsanity.com/paper/2602.04491/) (2026)
- [Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task](https://mlsanity.com/paper/2211.15022/) (2022)

## Abstract (arXiv)

With parallelizable attention networks, the neural Transformer is very fast to train. However, due to the auto-regressive architecture and self-attention in the decoder, the decoding procedure becomes slow. To alleviate this issue, we propose an average attention network as an alternative to the self-attention network in the decoder of the neural Transformer. The average attention network consists of two layers, with an average layer that models dependencies on previous positions and a gating layer that is stacked over the average layer to enhance the expressiveness of the proposed attention network. We apply this network on the decoder part of the neural Transformer to replace the original target-side self-attention model. With masking tricks and dynamic programming, our model enables the neural Transformer to decode sentences over four times faster than its original version with almost no loss in training time and translation performance. We conduct a series of experiments on WMT17 translation tasks, where on 6 different language pairs, we obtain robust and consistent speed-ups in decoding.