Scalable Transformers for Neural Machine Translation
Not on the map. Why?
Abstract (arXiv)
Transformer has been widely adopted in Neural Machine Translation (NMT) because of its large capacity and parallel training of sequence generation. However, the deployment of Transformer is challenging because different scenarios require models of different complexities and scales. Naively training multiple Transformers is redundant in terms of both computation and memory. In this paper, we propose a novel Scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters. Each sub-Transformer can be easily obtained by cropping the parameters of the largest Transformer. A three-stage training scheme is proposed to tackle the difficulty of training the Scalable Transformers, which introduces additional supervisions from word-level and sequence-level self-distillation. Extensive experiments were conducted on WMT EN-De and En-Fr to validate our proposed Scalable Transformers.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 4 Jun 2021.
Builds on
32 of 42 references.
References · 32
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
- 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
20 more
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2016Deep Networks with Stochastic Depthcited by 973
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2018Universal Transformerscited by 613
- 2017Convolutional Sequence to Sequence Learningcited by 550
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
- 2018Slimmable Neural Networkscited by 301
- 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
- 2018Scaling Neural Machine Translationcited by 204
- 2019The Evolved Transformercited by 180
- 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
- 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
- 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
- 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
- 2017Depthwise Separable Convolutions for Neural Machine Translationcited by 26
- 2019Multi-modality Latent Interaction Network for Visual Question Answeringcited by 23
- 2021RomeBERT: Robust Training of Multi-Exit BERTcited by 19
- 2017Weighted Transformer Network for Machine Translationcited by 16
- 2020Multi-Pass Transformer for Machine Translationcited by 5
Built on by
Citing papers per year: 2021: 1; 2022: 1.
2 papers.