Bag of Tricks for Optimizing Transformer Efficiency
Not on the map. Why?
Abstract (arXiv)
Improving Transformer efficiency has become increasingly attractive recently. A wide range of methods has been proposed, e.g., pruning, quantization, new architectures and etc. But these methods are either sophisticated in implementation or dependent on hardware. In this paper, we show that the efficiency of Transformer can be improved by combining some simple and hardware-agnostic methods, including tuning hyper-parameters, better design choices and training strategies. On the WMT news translation tasks, we improve the inference efficiency of a strong Transformer system by 3.80X on CPU and 2.52X on GPU. The code is publicly available at https://github.com/Lollipop321/mini-decoder-network.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 9 Sep 2021.
Builds on
18 of 24 references.
References · 18
- 2017Attention Is All You Needcited by 59,932
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
- 2017Non-Autoregressive Neural Machine Translationcited by 457
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2016Efficient softmax approximation for GPUscited by 67
- 2019Sharing Attention Weights for Fast Transformercited by 34
6 more
- 2020Rethinking the Value of Transformer Componentscited by 24
- 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
- 2018Accelerating Neural Transformer via an Average Attention Networkcited by 22
- 2020Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translationcited by 13
- 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
- 2020Efficient Inference For Neural Machine Translationcited by 7
Built on by
Citing papers per year: 2021: 1; 2022: 1; 2023: 1.
3 papers.