# Small Batch Sizes Improve Training of Low-Resource Neural MT

arXiv: https://arxiv.org/abs/2203.10579 · HTML: https://mlsanity.com/paper/2203.10579/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [OpenNMT: Neural Machine Translation Toolkit](https://mlsanity.com/paper/1805.11462/) (2018)
- [Optimizing Transformer for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.02266/) (2020)
- [On Optimal Transformer Depth for Low-Resource Language Translation](https://mlsanity.com/paper/2004.04418/) (2020)
- [Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change](https://mlsanity.com/paper/2005.02008/) (2020)
- [SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task](https://mlsanity.com/paper/2010.05122/) (2020)
- [CUNI Systems for the Unsupervised and Very Low Resource Translation Task in WMT20](https://mlsanity.com/paper/2010.11747/) (2020)

## Built on by (most cited first, 2 of 2)

- [Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Models](https://mlsanity.com/paper/2210.15184/) (2022)
- [Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset](https://mlsanity.com/paper/2303.12892/) (2023)

## Abstract (arXiv)

We study the role of an essential hyper-parameter that governs the training of Transformers for neural machine translation in a low-resource setting: the batch size. Using theoretical insights and experimental evidence, we argue against the widespread belief that batch size should be set as large as allowed by the memory of the GPUs. We show that in a low-resource setting, a smaller batch size leads to higher scores in a shorter training time, and argue that this is due to better regularization of the gradients during training.