# Large-Batch Training for LSTM and Beyond

arXiv: https://arxiv.org/abs/1901.08256 · HTML: https://mlsanity.com/paper/1901.08256/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2019.

## Built on by (most cited first, 21 of 21)

- [Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines](https://mlsanity.com/paper/2107.06925/) (2021)
- [Stochastic Training is Not Necessary for Generalization](https://mlsanity.com/paper/2109.14119/) (2021)
- [DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training](https://mlsanity.com/paper/2104.11981/) (2021)
- [Near-Optimal Sparse Allreduce for Distributed Deep Learning](https://mlsanity.com/paper/2201.07598/) (2022)
- [Concurrent Adversarial Learning for Large-Batch Training](https://mlsanity.com/paper/2106.00221/) (2021)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)
- [COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression](https://mlsanity.com/paper/2111.09562/) (2021)
- [Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters](https://mlsanity.com/paper/2108.03645/) (2021)
- [Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences](https://mlsanity.com/paper/2203.04738/) (2022)
- [Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules](https://mlsanity.com/paper/2207.11333/) (2022)
- [Uncertainty-Informed Volume Visualization using Implicit Neural Representation](https://mlsanity.com/paper/2408.06018/) (2024)
- [NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning](https://mlsanity.com/paper/2106.07454/) (2021)
- [Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity](https://mlsanity.com/paper/2201.12667/) (2022)
- [Visual Analysis of Prediction Uncertainty in Neural Networks for Deep Image Synthesis](https://mlsanity.com/paper/2406.18545/) (2024)
- [CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation](https://mlsanity.com/paper/2106.10796/) (2021)
- [Logit Attenuating Weight Normalization](https://mlsanity.com/paper/2108.05839/) (2021)
- [Optimizing Neural Network for Computer Vision task in Edge Device](https://mlsanity.com/paper/2110.00791/) (2021)
- [Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization](https://mlsanity.com/paper/2302.05865/) (2023)
- [Mastery Guided Non-parametric Clustering to Scale-up Strategy Prediction](https://mlsanity.com/paper/2401.10210/) (2024)
- [DHO2: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM](https://mlsanity.com/paper/2505.00982/) (2025)
- [Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training](https://mlsanity.com/paper/2603.18112/) (2026)

## Abstract (arXiv)

Large-batch training approaches have enabled researchers to utilize large-scale distributed processing and greatly accelerate deep-neural net (DNN) training. For example, by scaling the batch size from 256 to 32K, researchers have been able to reduce the training time of ResNet50 on ImageNet from 29 hours to 2.2 minutes (Ying et al., 2018). In this paper, we propose a new approach called linear-epoch gradual-warmup (LEGW) for better large-batch training. With LEGW, we are able to conduct large-batch training for both CNNs and RNNs with the Sqrt Scaling scheme. LEGW enables Sqrt Scaling scheme to be useful in practice and as a result we achieve much better results than the Linear Scaling learning rate scheme. For LSTM applications, we are able to scale the batch size by a factor of 64 without losing accuracy and without tuning the hyper-parameters. For CNN applications, LEGW is able to achieve the same accuracy even as we scale the batch size to 32K. LEGW works better than previous large-batch auto-tuning techniques. LEGW achieves a 5.3X average speedup over the baselines for four LSTM-based applications on the same hardware. We also provide some theoretical explanations for LEGW.