The NiuTrans System for the WMT21 Efficiency Task
Not on the map. Why?
Abstract (arXiv)
This paper describes the NiuTrans system for the WMT21 translation efficiency task (http://statmt.org/wmt21/efficiency-task.html). Following last year's work, we explore various techniques to improve efficiency while maintaining translation quality. We investigate the combinations of lightweight Transformer architectures and knowledge distillation strategies. Also, we improve the translation efficiency with graph optimization, low precision, dynamic batching, and parallel pre/post-processing. Our system can translate 247,000 words per second on an NVIDIA A100, being 3× faster than last year's system. Our system is the fastest and has the lowest memory consumption on the GPU-throughput track. The code, model, and pipeline will be available at NiuTrans.NMT (https://github.com/NiuTrans/NiuTrans.NMT).
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 16 Sep 2021.
Builds on
23 of 31 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2018Universal Transformerscited by 613
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
11 more
- 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
- 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
- 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 40
- 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
- 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
- 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
- 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
- 2021RankNAS: Efficient Neural Architecture Search by Pairwise Rankingcited by 12
- 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
- 2021Bag of Tricks for Optimizing Transformer Efficiencycited by 3
- 2020Training Flexible Depth Model by Multi-Task Learning for Neural Machine Translationcited by 1
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.