mlsanityData to 7 Oct 2026

Learning Deep Transformer Models for Machine Translation

arXiv 1906.01787 ↗First posted 5 Jun 2019cs.CL cs.LGcited by 309

Not on the map. Why?

Abstract (arXiv)

Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT'16 English- German, NIST OpenMT'12 Chinese-English and larger WMT'18 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big.

Key facts. Cited by 309 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 70; 2022: 60; 2023: 72; 2024: 50; 2025: 39; 2026: 17.

309 papers.

Sort:

By area:

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  3. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  4. 2021A Survey of Transformerscited by 399
  5. 2021Early Convolutions Help Transformers See Bettercited by 395
  6. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 342
  7. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  8. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  9. 2022Masked Visual Pre-training for Motor Controlcited by 246
  10. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  11. 2021Anticipative Video Transformercited by 197
  12. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
38 more
  1. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  2. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  3. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  4. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  5. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  6. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  7. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  8. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  9. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  10. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  11. 2021UTNet: A Hybrid Transformer Architecture for Medical Image Segmentationcited by 116
  12. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  13. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  14. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  15. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  16. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  17. 2021Luna: Linear Unified Nested Attentioncited by 96
  18. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  19. 2021Inpainting Transformer for Anomaly Detectioncited by 91
  20. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 87
  21. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  22. 2023ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Textcited by 84
  23. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  24. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 65
  25. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  26. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 58
  27. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58
  28. 2025The Curse of Depth in Large Language Modelscited by 58
  29. 2022PinnerFormer: Sequence Modeling for User Representation at Pinterestcited by 57
  30. 2023Spatial Functa: Scaling Functa to ImageNet Classification and Generationcited by 55
  31. 2023Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Seriescited by 54
  32. 2021SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinationscited by 51
  33. 2024MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereocited by 49
  34. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  35. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  36. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  37. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  38. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44

50 most cited of 309 All 309 (JSON)