mlsanityData to 7 Oct 2026

The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

arXiv 1804.09849 ↗First posted 26 Apr 2018cs.CL cs.AIcited by 101

Not on the map. Why?

Abstract (arXiv)

The past year has witnessed rapid advances in sequence-to-sequence (seq2seq) modeling for Machine Translation (MT). The classic RNN-based approaches to MT were first out-performed by the convolutional seq2seq model, which was then out-performed by the more recent Transformer model. Each of these new approaches consists of a fundamental architecture accompanied by a set of modeling and training techniques that are in principle applicable to other seq2seq architectures. In this paper, we tease apart the new architectures and their accompanying techniques in two ways. First, we identify several key modeling and training techniques, and apply them to the RNN architecture, yielding a new RNMT+ model that outperforms all of the three fundamental architectures on the benchmark WMT'14 English to French and English to German tasks. Second, we analyze the properties of each fundamental seq2seq architecture and devise new hybrid architectures intended to combine their strengths. Our hybrid models obtain further improvements, outperforming the RNMT+ model on both benchmark datasets.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 35; 2022: 32; 2023: 21; 2024: 7; 2025: 4; 2026: 1.

101 papers.

Sort:

By area:

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  4. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  5. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  6. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  7. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  8. 2021Scaling Laws for Neural Machine Translationcited by 110
  9. 2022Block-Recurrent Transformerscited by 107
  10. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  11. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  12. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
38 more
  1. 2021Space-time Mixing Attention for Video Transformercited by 84
  2. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  3. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  4. 2021Geometric Deep Learning and Equivariant Neural Networkscited by 64
  5. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  6. 2023Improved Operator Learning by Orthogonal Attentioncited by 58
  7. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  8. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  9. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  10. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  11. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  12. 2022Hardware Approximate Techniques for Deep Neural Network Accelerators: A Surveycited by 39
  13. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 36
  14. 2022DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalizationcited by 35
  15. 2023LLMRec: Benchmarking Large Language Models on Recommendation Taskcited by 29
  16. 2021Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimizationcited by 27
  17. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  18. 2022Examining Scaling and Transfer of Language Model Architectures for Machine Translationcited by 26
  19. 2021Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognitioncited by 24
  20. 2019Uncertainty-Aware Data Aggregation for Deep Imitation Learningcited by 19
  21. 2022Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learningcited by 19
  22. 2022Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Setscited by 17
  23. 2021Multi-head or Single-head? An Empirical Comparison for Transformer Trainingcited by 15
  24. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  25. 2021Diverse Pretrained Context Encodings Improve Document Translationcited by 13
  26. 2021Staircase Attention for Recurrent Processing of Sequencescited by 13
  27. 2022An Empirical Study on Activity Recognition in Long Surgical Videoscited by 13
  28. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  29. 2023Static Code Analysis in the AI Era: An In-depth Exploration of the Concept, Function, and Potential of Intelligent Code Analysis Agentscited by 12
  30. 2022Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restorationcited by 11
  31. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  32. 2023Linear Log-Normal Attention with Unbiased Concentrationcited by 11
  33. 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
  34. 2021Efficient Inference for Multilingual Neural Machine Translationcited by 10
  35. 2022GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translationcited by 10
  36. 2021A Masked Segmental Language Model for Unsupervised Natural Language Segmentationcited by 9
  37. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  38. 2024FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Drivingcited by 9

50 most cited of 101 All 101 (JSON)