Small Batch Sizes Improve Training of Low-Resource Neural MT
Not on the map. Why?
Abstract (arXiv)
We study the role of an essential hyper-parameter that governs the training of Transformers for neural machine translation in a low-resource setting: the batch size. Using theoretical insights and experimental evidence, we argue against the widespread belief that batch size should be set as large as allowed by the memory of the GPUs. We show that in a low-resource setting, a smaller batch size leads to higher scores in a shorter training time, and argue that this is due to better regularization of the gradients during training.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 20 Mar 2022.
Builds on
17 of 31 references.
References · 17
- 2017Attention Is All You Needcited by 59,932
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
- 2017Generalization in Deep Learningcited by 146
- 2018Training Tips for the Transformer Modelcited by 85
- 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
- 2018OpenNMT: Neural Machine Translation Toolkitcited by 27
5 more
- 2020Optimizing Transformer for Low-Resource Neural Machine Translationcited by 24
- 2020On Optimal Transformer Depth for Low-Resource Language Translationcited by 9
- 2020Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Changecited by 4
- 2020SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Taskcited by 4
- 2020CUNI Systems for the Unsupervised and Very Low Resource Translation Task in WMT20cited by 2
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.