mlsanityData to 14 Oct 2026

Training Tips for the Transformer Model

arXiv 1804.00247 ↗First posted 1 Apr 2018cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

This article describes our experiments in neural machine translation using the recent Tensor2Tensor framework and the Transformer sequence-to-sequence model (Vaswani et al., 2017). We examine some of the critical parameters that affect the final translation quality, memory usage, training stability and training time, concluding each experiment with a set of recommendations for fellow researchers. In addition to confirming the general mantra "more data and larger models", we address scaling to multiple GPUs and provide practical tips for improved training regarding batch size, learning rate, warmup steps, maximum sentence length and checkpoint averaging. We hope that our observations will allow others to get better results given their particular hardware and data constraints.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 21; 2022: 18; 2023: 22; 2024: 15; 2025: 8; 2026: 1.

85 papers.

Sort:

By area:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  3. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  4. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  5. 2021Distributed Deep Learning in Open Collaborationscited by 58
  6. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
  7. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  8. 2022Transformers in Time-series Analysis: A Tutorialcited by 46
  9. 2021CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generationcited by 42
  10. 2021Prototypical Representation Learning for Relation Extractioncited by 33
  11. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
  12. 2021Why don't people use character-level machine translation?cited by 27
38 more
  1. 2021Persformer: A Transformer Architecture for Topological Machine Learningcited by 19
  2. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  3. 2022Distributionally Robust Models with Parametric Likelihood Ratioscited by 17
  4. 2022Hausa Visual Genome: A Dataset for Multi-Modal English to Hausa Machine Translationcited by 13
  5. 2022Revisiting Checkpoint Averaging for Neural Machine Translationcited by 13
  6. 2021Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Modelscited by 12
  7. 2021Transformers: "The End of History" for NLP?cited by 12
  8. 2023HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Languagecited by 12
  9. 2021Extended Parallel Corpus for Amharic-English Machine Translationcited by 10
  10. 2021Contextual Semantic Parsing for Multilingual Task-Oriented Dialoguescited by 10
  11. 2022AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modelingcited by 10
  12. 2024EPi-cKANs: Elasto-Plasticity Informed Kolmogorov-Arnold Networks Using Chebyshev Polynomialscited by 9
  13. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  14. 2021GTNet:Guided Transformer Network for Detecting Human-Object Interactionscited by 8
  15. 2022Compositionality as Lexical Symmetrycited by 8
  16. 2023Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMscited by 8
  17. 2023Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averagingcited by 8
  18. 2022Focused Concatenation for Context-Aware Neural Machine Translationcited by 7
  19. 2021Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classificationcited by 6
  20. 2021Rethinking Adam: A Twofold Exponential Moving Average Approachcited by 6
  21. 2023Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenationcited by 6
  22. 2023Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizerscited by 6
  23. 2022Training Transformers Togethercited by 5
  24. 2022Relaxed Attention for Transformer Modelscited by 5
  25. 2023eWaSR -- an embedded-compute-ready maritime obstacle detection networkcited by 5
  26. 2023SkinDistilViT: Lightweight Vision Transformer for Skin Lesion Classificationcited by 5
  27. 2024Frequency matters: Modeling irregular morphological patterns in Spanish with Transformerscited by 5
  28. 2025Stability of Transformers under Layer Normalizationcited by 5
  29. 2022Gradient Descent, Stochastic Optimization, and Other Talescited by 4
  30. 2024A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learningcited by 4
  31. 2024The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraintscited by 4
  32. 2021Text Simplification for Comprehension-based Question-Answeringcited by 3
  33. 2022Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22cited by 3
  34. 2023Curriculum-guided Abstractive Summarization for Mental Health Online Postscited by 3
  35. 2023NL2CMD: An Updated Workflow for Natural Language to Bash Commands Translationcited by 3
  36. 2023Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Datasetcited by 3
  37. 2023A Neural Network Transformer Model for Composite Microstructure Homogenizationcited by 3
  38. 2024When Will Gradient Regularization Be Harmful?cited by 3

50 most cited of 85 All 85 (JSON)