# Training Generative Adversarial Networks by Solving Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2010.15040 · HTML: https://mlsanity.com/paper/2010.15040/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher](https://mlsanity.com/paper/2405.14822/) (2024)
- [Which priors matter? Benchmarking models for learning latent dynamics](https://mlsanity.com/paper/2111.05458/) (2021)
- [Discretization Drift in Two-Player Games](https://mlsanity.com/paper/2105.13922/) (2021)
- [SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer](https://mlsanity.com/paper/2301.12811/) (2023)
- [Complex Momentum for Optimization in Games](https://mlsanity.com/paper/2102.08431/) (2021)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [Self-Destructive Language Model](https://mlsanity.com/paper/2505.12186/) (2025)
- [Hyperspectral Image Classification With Contrastive Graph Convolutional Network](https://mlsanity.com/paper/2205.11237/) (2022)
- [The Geometric Occam's Razor Implicit in Deep Learning](https://mlsanity.com/paper/2111.15090/) (2021)
- [Domain Adversarial Training: A Game Perspective](https://mlsanity.com/paper/2202.05352/) (2022)
- [SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision](https://mlsanity.com/paper/2111.05986/) (2021)
- [Functional Space Analysis of Local GAN Convergence](https://mlsanity.com/paper/2102.04448/) (2021)
- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)
- [Recursive Reasoning in Minimax Games: A Level k Gradient Play Method](https://mlsanity.com/paper/2210.16482/) (2022)
- [Training Generative Adversarial Networks with Adaptive Composite Gradient](https://mlsanity.com/paper/2111.05508/) (2021)
- [3D GANs and Latent Space: A comprehensive survey](https://mlsanity.com/paper/2304.03932/) (2023)
- [Control of Overfitting with Physics](https://mlsanity.com/paper/2412.10716/) (2024)
- [Learning by solving differential equations](https://mlsanity.com/paper/2505.13397/) (2025)
- [A comprehensive analysis of PINNs: Variants, Applications, and Challenges](https://mlsanity.com/paper/2505.22761/) (2025)
- [On the Nash equilibrium of moment-matching GANs for stationary Gaussian processes](https://mlsanity.com/paper/2203.07136/) (2022)

## Abstract (arXiv)

The instability of Generative Adversarial Network (GAN) training has frequently been attributed to gradient descent. Consequently, recent methods have aimed to tailor the models and training procedures to stabilise the discrete updates. In contrast, we study the continuous-time dynamics induced by GAN training. Both theory and toy experiments suggest that these dynamics are in fact surprisingly stable. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error in discretising the continuous dynamics. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training - when combined with a regulariser that controls the integration error. Our approach represents a radical departure from previous methods which typically use adaptive optimisation and stabilisation techniques that constrain the functional space (e.g. Spectral Normalisation). Evaluation on CIFAR-10 and ImageNet shows that our method outperforms several strong baselines, demonstrating its efficacy.