mlsanityData to 14 Oct 2026

Scaling Neural Machine Translation

arXiv 1806.00187 ↗First posted 1 Jun 2018cs.CLcited by 204

Not on the map. Why?

Abstract (arXiv)

Sequence to sequence learning models still require several days to reach state of the art performance on large benchmark datasets using a single machine. This paper shows that reduced precision and large batch training can speedup training by nearly 5x on a single 8-GPU machine with careful tuning and implementation. On WMT'14 English-German translation, we match the accuracy of Vaswani et al. (2017) in under 5 hours when training on 8 GPUs and we obtain a new state of the art of 29.3 BLEU after training for 85 minutes on 128 GPUs. We further improve these results to 29.8 BLEU by training on the much larger Paracrawl dataset. On the WMT'14 English-French task, we obtain a state-of-the-art BLEU of 43.2 in 8.5 hours on 128 GPUs.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 91; 2022: 52; 2023: 32; 2024: 11; 2025: 13; 2026: 4.

204 papers.

Sort:

By area:

  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2021Transformers in Vision: A Surveycited by 1,013
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  6. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  8. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  9. 2021Random Feature Attentioncited by 354
  10. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 339
  11. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 320
  12. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
38 more
  1. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  2. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  3. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  4. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  5. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  7. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  9. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  10. 2021Entailment as Few-Shot Learnercited by 123
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  12. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  13. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  14. 2021Finetuning Pretrained Transformers into RNNscited by 96
  15. 2021Luna: Linear Unified Nested Attentioncited by 96
  16. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  17. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  18. 2021How to Train BERT with an Academic Budgetcited by 80
  19. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  20. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  21. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  22. 2021Bad Characters: Imperceptible NLP Attackscited by 71
  23. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  24. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 60
  25. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 59
  26. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  27. 2021Distributed Deep Learning in Open Collaborationscited by 58
  28. 2021TransZero: Attribute-guided Transformer for Zero-Shot Learningcited by 54
  29. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 52
  30. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  31. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  32. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  33. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  34. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  35. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  36. 2021Video Transformer for Deepfake Detection with Incremental Learningcited by 33
  37. 2022Reversible Column Networkscited by 32
  38. 2023Sparse Universal Transformercited by 32

50 most cited of 204 All 204 (JSON)