# Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule

arXiv: https://arxiv.org/abs/2201.11989 · HTML: https://mlsanity.com/paper/2201.11989/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Measuring the Effects of Data Parallelism on Neural Network Training](https://mlsanity.com/paper/1811.03600/) (2018)
- [On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization](https://mlsanity.com/paper/1808.02941/) (2018)
- [Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model](https://mlsanity.com/paper/1907.04164/) (2019)
- [COT-GAN: Generating Sequential Data via Causal Optimal Transport](https://mlsanity.com/paper/2006.08571/) (2020)
- [Gradient descent GAN optimization is locally stable](https://mlsanity.com/paper/1706.04156/) (2017)
- [Convergence rates for the stochastic gradient descent method for non-convex objective functions](https://mlsanity.com/paper/1904.01517/) (2019)
- [Reducing Noise in GAN Training with Variance Reduced Extragradient](https://mlsanity.com/paper/1904.08598/) (2019)
- [Low-Rank Subspaces in GANs](https://mlsanity.com/paper/2106.04488/) (2021)
- [Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks](https://mlsanity.com/paper/2002.09647/) (2020)
- [Progressive Augmentation of GANs](https://mlsanity.com/paper/1901.10422/) (2019)
- [Robust conditional GANs under missing or uncertain labels](https://mlsanity.com/paper/1906.03579/) (2019)

## Built on by (most cited first, 8 of 8)

- [Convergence Bound and Critical Batch Size of Muon Optimizer](https://mlsanity.com/paper/2507.01598/) (2025)
- [Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Rates](https://mlsanity.com/paper/2402.15344/) (2024)
- [Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2501.08883/) (2025)
- [Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size](https://mlsanity.com/paper/2501.18164/) (2025)
- [Both Asymptotic and Non-Asymptotic Convergence of Quasi-Hyperbolic Momentum using Increasing Batch Size](https://mlsanity.com/paper/2506.23544/) (2025)
- [Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis](https://mlsanity.com/paper/2508.03105/) (2025)
- [Relationship between Batch Size and Number of Steps Needed for Nonconvex Optimization of Stochastic Gradient Descent using Armijo Line Search](https://mlsanity.com/paper/2307.13831/) (2023)
- [Adaptive Batch Size and Learning Rate Scheduler for Stochastic Gradient Descent Based on Minimization of Stochastic First-order Oracle Complexity](https://mlsanity.com/paper/2508.05302/) (2025)

## Abstract (arXiv)

Previous results have shown that a two time-scale update rule (TTUR) using different learning rates, such as different constant rates or different decaying rates, is useful for training generative adversarial networks (GANs) in theory and in practice. Moreover, not only the learning rate but also the batch size is important for training GANs with TTURs and they both affect the number of steps needed for training. This paper studies the relationship between batch size and the number of steps needed for training GANs with TTURs based on constant learning rates. We theoretically show that, for a TTUR with constant learning rates, the number of steps needed to find stationary points of the loss functions of both the discriminator and generator decreases as the batch size increases and that there exists a critical batch size minimizing the stochastic first-order oracle (SFO) complexity. Then, we use the Fr'echet inception distance (FID) as the performance measure for training and provide numerical results indicating that the number of steps needed to achieve a low FID score decreases as the batch size increases and that the SFO complexity increases once the batch size exceeds the measured critical batch size. Moreover, we show that measured critical batch sizes are close to the sizes estimated from our theoretical results.