# CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers

arXiv: https://arxiv.org/abs/1901.02244 · HTML: https://mlsanity.com/paper/1901.02244/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2019.

## Built on by (most cited first, 14 of 14)

- [TorchOpt: An Efficient Library for Differentiable Optimization](https://mlsanity.com/paper/2211.06934/) (2022)
- [Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity](https://mlsanity.com/paper/2309.10285/) (2023)
- [The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey](https://mlsanity.com/paper/2305.13854/) (2023)
- [MegBA: A GPU-Based Distributed Library for Large-Scale Bundle Adjustment](https://mlsanity.com/paper/2112.01349/) (2021)
- [Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awareness](https://mlsanity.com/paper/2305.10863/) (2023)
- [An Analysis of Collocation on GPUs for Deep Learning Training](https://mlsanity.com/paper/2209.06018/) (2022)
- [Fast and Flexible Human Pose Estimation with HyperPose](https://mlsanity.com/paper/2108.11826/) (2021)
- [GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Models](https://mlsanity.com/paper/2310.05205/) (2023)
- [TensorSocket: Shared Data Loading for Deep Learning Training](https://mlsanity.com/paper/2409.18749/) (2024)
- [Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers](https://mlsanity.com/paper/2110.07029/) (2021)
- [A Survey on Video Analytics in Cloud-Edge-Terminal Collaborative Systems](https://mlsanity.com/paper/2502.06581/) (2025)
- [ByteComp: Revisiting Gradient Compression in Distributed Training](https://mlsanity.com/paper/2205.14465/) (2022)
- [Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks](https://mlsanity.com/paper/2304.08925/) (2023)
- [Reducing Fragmentation and Starvation in GPU Clusters through Dynamic Multi-Objective Scheduling](https://mlsanity.com/paper/2512.10980/) (2025)

## Abstract (arXiv)

Deep learning models are trained on servers with many GPUs, and training must scale with the number of GPUs. Systems such as TensorFlow and Caffe2 train models with parallel synchronous stochastic gradient descent: they process a batch of training data at a time, partitioned across GPUs, and average the resulting partial gradients to obtain an updated global model. To fully utilise all GPUs, systems must increase the batch size, which hinders statistical efficiency. Users tune hyper-parameters such as the learning rate to compensate for this, which is complex and model-specific. We describe CROSSBOW, a new single-server multi-GPU system for training deep learning models that enables users to freely choose their preferred batch size - however small - while scaling to multiple GPUs. CROSSBOW uses many parallel model replicas and avoids reduced statistical efficiency through a new synchronous training method. We introduce SMA, a synchronous variant of model averaging in which replicas independently explore the solution space with gradient descent, but adjust their search synchronously based on the trajectory of a globally-consistent average model. CROSSBOW achieves high hardware efficiency with small batch sizes by potentially training multiple model replicas per GPU, automatically tuning the number of replicas to maximise throughput. Our experiments show that CROSSBOW improves the training time of deep learning models on an 8-GPU server by 1.3-4x compared to TensorFlow.