mlsanityData to 7 Oct 2026

Very Deep Transformers for Neural Machine Translation

arXiv 2008.07772 ↗First posted 18 Aug 2020cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we show that it is feasible to build standard Transformer-based models with up to 60 encoder layers and 12 decoder layers. These deep models outperform their baseline 6-layer counterparts by as much as 2.5 BLEU, and achieve new state-of-the-art benchmark results on WMT14 English-French (43.8 BLEU and 46.4 BLEU with back-translation) and WMT14 English-German (30.1 BLEU).The code and trained models will be publicly available at: https://github.com/namisan/exdeep-nmt.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 14; 2023: 17; 2024: 3; 2025: 3.

50 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  3. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  4. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  5. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  6. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  7. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  8. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  9. 2022Grammatical Error Correction: A Survey of the State of the Artcited by 43
  10. 2021A Simple and Effective Positional Encoding for Transformerscited by 37
  11. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  12. 2023Sparse Universal Transformercited by 28
38 more
  1. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  2. 2023UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human languagecited by 22
  3. 2022Cedille: A large autoregressive French language modelcited by 20
  4. 2021Controlling Neural Networks with Rule Representationscited by 16
  5. 2021The King is Naked: on the Notion of Robustness for Natural Language Processingcited by 16
  6. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  7. 2023CLIP-QDA: An Explainable Concept Bottleneck Modelcited by 13
  8. 2021Self-supervised and Supervised Joint Training for Resource-rich Machine Translationcited by 12
  9. 2021WeChat Neural Machine Translation Systems for WMT21cited by 11
  10. 2022GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translationcited by 10
  11. 2022Variational Inference for Infinitely Deep Neural Networkscited by 10
  12. 2025Impact of Latent Space Dimension on IoT Botnet Detection Performance: VAE-Encoder Versus ViT-Encodercited by 6
  13. 2022Mental Illness Classification on Social Media Texts using Deep Learning and Transfer Learningcited by 5
  14. 2022Structural Biases for Improving Transformers on Translation into Morphologically Rich Languagescited by 5
  15. 2024Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translationcited by 5
  16. 2025IoT Botnet Detection: Application of Vision Transformer to Classification of Network Flow Trafficcited by 5
  17. 2023Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Designcited by 4
  18. 2023Core Building Blocks: Next Gen Geo Spatial GPT Applicationcited by 4
  19. 2021ZeBRA: Precisely Destroying Neural Networks with Zero-Data Based Repeated Bit Flip Attackcited by 3
  20. 2023Understanding Parameter Sharing in Transformerscited by 3
  21. 2022FAT: An In-Memory Accelerator with Fast Addition for Ternary Weight Neural Networkscited by 2
  22. 2022Emergent Linguistic Structures in Neural Networks are Fragilecited by 2
  23. 2024TransAdapter: Vision Transformer for Feature-Centric Unsupervised Domain Adaptationcited by 2
  24. 2022Isomorphic Cross-lingual Embeddings for Low-Resource Languagescited by 1
  25. 2022EIT: Enhanced Interactive Transformercited by 1
  26. 2023An Evaluation of Persian-English Machine Translation Datasets with Transformerscited by 1
  27. 2023MobileNMT: Enabling Translation in 15MB and 30mscited by 1
  28. 2023Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translationcited by 1
  29. 2021Cascaded Head-colliding Attentioncited by 0
  30. 2021Text analysis and deep learning: A network approachcited by 0
  31. 2022Towards Personalized Intelligence at Scalecited by 0
  32. 2022Why Robust Natural Language Understanding is a Challengecited by 0
  33. 2022Effective General-Domain Data Inclusion for the Machine Translation Task by Vanilla Transformerscited by 0
  34. 2023Machine Learning for Brain Disorders: Transformers and Visual Transformerscited by 0
  35. 2023Multi-Path Transformer is Better: A Case Study on Neural Machine Translationcited by 0
  36. 2023Enhanced Transformer Architecture for Natural Language Processingcited by 0
  37. 2024Enhancing Context Through Contrastcited by 0
  38. 2025Self-Vocabularizing Training for Neural Machine Translationcited by 0