# Bag of Tricks for Optimizing Transformer Efficiency

arXiv: https://arxiv.org/abs/2109.04030 · HTML: https://mlsanity.com/paper/2109.04030/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [Sharing Attention Weights for Fast Transformer](https://mlsanity.com/paper/1906.11024/) (2019)
- [Rethinking the Value of Transformer Components](https://mlsanity.com/paper/2011.03803/) (2020)
- [Weight Distillation: Transferring the Knowledge in Neural Network Parameters](https://mlsanity.com/paper/2009.09152/) (2020)
- [Accelerating Neural Transformer via an Average Attention Network](https://mlsanity.com/paper/1805.00631/) (2018)
- [Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation](https://mlsanity.com/paper/2009.07453/) (2020)
- [An Efficient Transformer Decoder with Compressed Sub-layers](https://mlsanity.com/paper/2101.00542/) (2021)
- [Efficient Inference For Neural Machine Translation](https://mlsanity.com/paper/2010.02416/) (2020)

## Built on by (most cited first, 3 of 3)

- [The NiuTrans System for the WMT21 Efficiency Task](https://mlsanity.com/paper/2109.08003/) (2021)
- [Building an Efficiency Pipeline: Commutativity and Cumulativeness of Efficiency Operators for Transformers](https://mlsanity.com/paper/2208.00483/) (2022)
- [MobileNMT: Enabling Translation in 15MB and 30ms](https://mlsanity.com/paper/2306.04235/) (2023)

## Abstract (arXiv)

Improving Transformer efficiency has become increasingly attractive recently. A wide range of methods has been proposed, e.g., pruning, quantization, new architectures and etc. But these methods are either sophisticated in implementation or dependent on hardware. In this paper, we show that the efficiency of Transformer can be improved by combining some simple and hardware-agnostic methods, including tuning hyper-parameters, better design choices and training strategies. On the WMT news translation tasks, we improve the inference efficiency of a strong Transformer system by 3.80X on CPU and 2.52X on GPU. The code is publicly available at https://github.com/Lollipop321/mini-decoder-network.