mlsanityData to 14 Oct 2026

Shallow-to-Deep Training for Neural Machine Translation

arXiv 2010.03737 ↗First posted 8 Oct 2020cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Deep encoders have been proven to be effective in improving neural machine translation (NMT) systems, but training an extremely deep encoder is time consuming. Moreover, why deep models help NMT is an open question. In this paper, we investigate the behavior of a well-tuned deep Transformer system. We find that stacking layers is helpful in improving the representation ability of NMT models and adjacent layers perform similarly. This inspires us to develop a shallow-to-deep training method that learns deep models by stacking shallow models. In this way, we successfully train a Transformer system with a 54-layer encoder. Experimental results on WMT'16 English-German and WMT'14 English-French translation tasks show that it is 1.4 × faster than training from scratch, and achieves a BLEU score of 30.33 and 43.29 on two tasks. The code is publicly available at https://github.com/libeineu/SDT-Training/.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 5; 2024: 2; 2025: 2; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  2. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  3. 2022Staged Training for Transformer Language Modelscited by 49
  4. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 37
  5. 2021Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Taskcited by 34
  6. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 27
  7. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  8. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  9. 2021Learning High-Dimensional Parametric Maps via Reduced Basis Adaptive Residual Networkscited by 16
  10. 2021ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translationcited by 12
  11. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
  12. 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
24 more
  1. 2021Efficient Inference for Multilingual Neural Machine Translationcited by 10
  2. 2022GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translationcited by 10
  3. 2023Towards Understanding and Improving Knowledge Distillation for Neural Machine Translationcited by 10
  4. 2022Learning Multiscale Transformer Models for Sequence Generationcited by 7
  5. 2021TopicRefine: Joint Topic Prediction and Dialogue Response Generation for Multi-turn End-to-End Dialogue Systemcited by 5
  6. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 5
  7. 2021The NiuTrans Machine Translation Systems for WMT21cited by 4
  8. 2022The Emotion is Not One-hot Encoding: Learning with Grayscale Label for Emotion Recognition in Conversationcited by 4
  9. 2022DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decodingcited by 4
  10. 2024Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learningcited by 4
  11. 2021Joint-training on Symbiosis Networks for Deep Nueral Machine Translation modelscited by 3
  12. 2022The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Taskcited by 3
  13. 2023Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalizationcited by 3
  14. 2021Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Spacecited by 2
  15. 2021The NiuTrans System for the WMT21 Efficiency Taskcited by 2
  16. 2023Layer-wise Representation Fusion for Compositional Generalizationcited by 2
  17. 2026SpanNorm: Reconciling Training Stability and Performance in Deep Transformerscited by 2
  18. 2021The NiuTrans End-to-End Speech Translation System for IWSLT 2021 Offline Taskcited by 1
  19. 2022EIT: Enhanced Interactive Transformercited by 1
  20. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  21. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  22. 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
  23. 2022Optimizing Deep Transformers for Chinese-Thai Low-Resource Translationcited by 0
  24. 2025ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameterscited by 0