# Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equations arXiv: https://arxiv.org/abs/2112.13826 ยท HTML: https://mlsanity.com/paper/2112.13826/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 74 references that are in the map. - First posted on arXiv 27 Dec 2021. ## Builds on (most cited first, 29 of 29) - [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018) - [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019) - [Training GANs with Optimism](https://mlsanity.com/paper/1711.00141/) (2017) - [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015) - [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017) - [A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach](https://mlsanity.com/paper/1901.08511/) (2019) - [Cycles in adversarial regularized learning](https://mlsanity.com/paper/1709.02738/) (2017) - [A Variational Perspective on Accelerated Methods in Optimization](https://mlsanity.com/paper/1603.04245/) (2016) - [The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization](https://mlsanity.com/paper/1807.03907/) (2018) - [Efficient Methods for Structured Nonconvex-Nonconcave Min-Max Optimization](https://mlsanity.com/paper/2011.00364/) (2020) - [On the convergence of single-call stochastic extra-gradient methods](https://mlsanity.com/paper/1908.08465/) (2019) - [Understanding the Acceleration Phenomenon via High-Resolution Differential Equations](https://mlsanity.com/paper/1810.08907/) (2018) - [Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks](https://mlsanity.com/paper/1802.06132/) (2018) - [Reducing Noise in GAN Training with Variance Reduced Extragradient](https://mlsanity.com/paper/1904.08598/) (2019) - [Negative Momentum for Improved Game Dynamics](https://mlsanity.com/paper/1807.04740/) (2018) - [Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems](https://mlsanity.com/paper/2002.00057/) (2020) - [The limits of min-max optimization algorithms: convergence to spurious non-critical sets](https://mlsanity.com/paper/2006.09065/) (2020) - [Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling](https://mlsanity.com/paper/2003.10162/) (2020) - [Tight last-iterate convergence rates for no-regret learning in multi-player games](https://mlsanity.com/paper/2010.13724/) (2020) - [On Gradient-Based Learning in Continuous Games](https://mlsanity.com/paper/1804.05464/) (2018) - [On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach](https://mlsanity.com/paper/1910.07512/) (2019) - [Stochastic Hamiltonian Gradient Methods for Smooth Games](https://mlsanity.com/paper/2007.04202/) (2020) - [Local Saddle Point Optimization: A Curvature Exploitation Approach](https://mlsanity.com/paper/1805.05751/) (2018) - [Last-iterate convergence rates for min-max optimization](https://mlsanity.com/paper/1906.02027/) (2019) - [A Closer Look at the Optimization Landscapes of Generative Adversarial Networks](https://mlsanity.com/paper/1906.04848/) (2019) - [On dissipative symplectic integration with applications to gradient-based optimization](https://mlsanity.com/paper/2004.06840/) (2020) - [Taming GANs with Lookahead-Minmax](https://mlsanity.com/paper/2006.14567/) (2020) - [On the Iteration Complexity of Oblivious First-Order Optimization Algorithms](https://mlsanity.com/paper/1605.03529/) (2016) - [A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraints](https://mlsanity.com/paper/2009.11359/) (2020) ## Built on by (most cited first, 8 of 8) - [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023) - [Solving Constrained Variational Inequalities via a First-order Interior Point-based Method](https://mlsanity.com/paper/2206.10575/) (2022) - [Continuous-time Analysis for Variational Inequalities: An Overview and Desiderata](https://mlsanity.com/paper/2207.07105/) (2022) - [Extragradient Type Methods for Riemannian Variational Inequality Problems](https://mlsanity.com/paper/2309.14155/) (2023) - [Learning Variational Inequalities from Data: Fast Generalization Rates under Strong Monotonicity](https://mlsanity.com/paper/2410.20649/) (2024) - [A Gentle Introduction to Gradient-Based Optimization and Variational Inequalities for Machine Learning](https://mlsanity.com/paper/2309.04877/) (2023) - [Understanding Lookahead Dynamics Through Laplace Transform](https://mlsanity.com/paper/2506.13712/) (2025) - [Frequency-Based Hyperparameter Selection in Games](https://mlsanity.com/paper/2601.18409/) (2026) ## Abstract (arXiv) Several widely-used first-order saddle-point optimization methods yield an identical continuous-time ordinary differential equation (ODE) that is identical to that of the Gradient Descent Ascent (GDA) method when derived naively. However, the convergence properties of these methods are qualitatively different, even on simple bilinear games. Thus the ODE perspective, which has proved powerful in analyzing single-objective optimization methods, has not played a similar role in saddle-point optimization. We adopt a framework studied in fluid dynamics -- known as High-Resolution Differential Equations (HRDEs) -- to design differential equation models for several saddle-point optimization methods. Critically, these HRDEs are distinct for various saddle-point optimization methods. Moreover, in bilinear games, the convergence properties of the HRDEs match the qualitative features of the corresponding discrete methods. Additionally, we show that the HRDE of Optimistic Gradient Descent Ascent (OGDA) exhibits last-iterate convergence for general monotone variational inequalities. Finally, we provide rates of convergence for the best-iterate convergence of the OGDA method, relying solely on the first-order smoothness of the monotone operator.