# Augment your batch: better training with larger batches

arXiv: https://arxiv.org/abs/1901.09335 · HTML: https://mlsanity.com/paper/1901.09335/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2019.

## Built on by (most cited first, 30 of 37)

- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [Corrupted Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2202.03382/) (2022)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [Galileo: Learning Global & Local Features of Many Remote Sensing Modalities](https://mlsanity.com/paper/2502.09356/) (2025)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://mlsanity.com/paper/2105.13343/) (2021)
- [Improving Pixel-based MIM by Reducing Wasted Modeling Capability](https://mlsanity.com/paper/2308.00261/) (2023)
- [Differentially Private Image Classification by Learning Priors from Random Processes](https://mlsanity.com/paper/2306.06076/) (2023)
- [Adaptive Contrast for Image Regression in Computer-Aided Disease Assessment](https://mlsanity.com/paper/2112.11700/) (2021)
- [CVNets: High Performance Library for Computer Vision](https://mlsanity.com/paper/2206.02002/) (2022)
- [Unlocking Accuracy and Fairness in Differentially Private Image Classification](https://mlsanity.com/paper/2308.10888/) (2023)
- [ConvNets Match Vision Transformers at Scale](https://mlsanity.com/paper/2310.16764/) (2023)
- [Turbo Training with Token Dropout](https://mlsanity.com/paper/2210.04889/) (2022)
- [A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization](https://mlsanity.com/paper/2212.04486/) (2022)
- [Correlational Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2303.12670/) (2023)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Poly-View Contrastive Learning](https://mlsanity.com/paper/2403.05490/) (2024)
- [ShapeAug: Occlusion Augmentation for Event Camera Data](https://mlsanity.com/paper/2401.02274/) (2024)
- [FocusFormer: Focusing on What We Need via Architecture Sampler](https://mlsanity.com/paper/2208.10861/) (2022)
- [Do Your Best and Get Enough Rest for Continual Learning](https://mlsanity.com/paper/2503.18371/) (2025)
- [CodeFort: Robust Training for Code Generation Models](https://mlsanity.com/paper/2405.01567/) (2024)
- [Differentially Private Clustered Federated Learning](https://mlsanity.com/paper/2405.19272/) (2024)
- [SmoothNets: Optimizing CNN architecture design for differentially private deep learning](https://mlsanity.com/paper/2205.04095/) (2022)
- [Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)](https://mlsanity.com/paper/2309.13681/) (2023)
- [Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networks](https://mlsanity.com/paper/2509.24886/) (2025)
- [Beyond Classification: Knowledge Distillation using Multi-Object Impressions](https://mlsanity.com/paper/2110.14215/) (2021)

## Abstract (arXiv)

Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.