mlsanityData to 7 Oct 2026

Training Deeper Neural Machine Translation Models with Transparent Attention

arXiv 1808.07561 ↗First posted 22 Aug 2018cs.CL cs.AI cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 10; 2022: 10; 2023: 5; 2024: 2; 2025: 3; 2026: 6.

37 papers.

Sort:

By area:

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2022Transformers in Time Series: A Surveycited by 489
  3. 2021A Survey of Transformerscited by 399
  4. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  5. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  6. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  7. 2025The Curse of Depth in Large Language Modelscited by 58
  8. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  9. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  10. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  11. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  12. 2021Why don't people use character-level machine translation?cited by 27
25 more
  1. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 26
  2. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  3. 2023Cross-Layer Retrospective Retrieving via Layer Attentioncited by 20
  4. 2022B2T Connection: Serving Stability and Performance in Deep Transformerscited by 16
  5. 2021ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translationcited by 12
  6. 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
  7. 2022GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translationcited by 10
  8. 2022Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classificationcited by 9
  9. 2024Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokenscited by 6
  10. 2022Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attentioncited by 4
  11. 2025You Do Not Fully Utilize Transformer's Representation Capacitycited by 4
  12. 2021Recurrent multiple shared layers in Depth for Neural Machine Translationcited by 3
  13. 2023Ask Language Model to Clean Your Noisy Translation Datacited by 3
  14. 2023Layer-wise Representation Fusion for Compositional Generalizationcited by 2
  15. 2021Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translationcited by 1
  16. 2023Prompt Guided Copy Mechanism for Conversational Question Answeringcited by 1
  17. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  18. 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
  19. 2026ClaimFlow: Tracing the Evolution of Scientific Claims in NLPcited by 1
  20. 2026Do Value Vectors in Deep Layers Need Context from the Residual Stream?cited by 1
  21. 2022Optimizing Deep Transformers for Chinese-Thai Low-Resource Translationcited by 0
  22. 2025Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languagescited by 0
  23. 2026HAARES Half-Split Residual Basis Routing for Deep Transformerscited by 0
  24. 2026Dual Attention Residualscited by 0
  25. 2026Dynamic Parameterization Is Not Dynamic Inferencecited by 0