mlsanityData to 7 Oct 2026

A Gentle Introduction to Gradient-Based Optimization and Variational Inequalities for Machine Learning

arXiv 2309.04877 ↗First posted 9 Sep 2023cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The rapid progress in machine learning in recent years has been based on a highly productive connection to gradient-based optimization. Further progress hinges in part on a shift in focus from pattern recognition to decision-making and multi-agent problems. In these broader settings, new mathematical challenges emerge that involve equilibria and game theory instead of optima. Gradient-based methods remain essential -- given the high dimensionality and large scale of machine-learning problems -- but simple gradient descent is no longer the point of departure for algorithm design. We provide a gentle introduction to a broader framework for gradient-based algorithms in machine learning, beginning with saddle points and monotone games, and proceeding to general variational inequalities. While we provide convergence proofs for several of the algorithms that we present, our main focus is that of providing motivation and intuition.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 51 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

22 of 51 references.

References · 22

  1. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  2. 2020Performative Predictioncited by 318
  3. 2017How to Escape Saddle Points Efficientlycited by 284
  4. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  5. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  7. 2017Training GANs with Optimismcited by 217
  8. 2014Theoretical guarantees for approximate sampling from smooth and log-concave densitiescited by 206
  9. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  10. 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 144
  11. 2019A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approachcited by 142
  12. 2017Underdamped Langevin MCMC: A non-asymptotic analysiscited by 131
10 more
  1. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  2. 2017Convergence of Langevin MCMC in KL-divergencecited by 86
  3. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  4. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  5. 2021Who Leads and Who Follows in Strategic Classification?cited by 53
  6. 2019High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithmcited by 36
  7. 2019On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Gamescited by 35
  8. 2019Langevin Monte Carlo without smoothnesscited by 27
  9. 2020On dissipative symplectic integration with applications to gradient-based optimizationcited by 23
  10. 2021Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equationscited by 8

Built on by

Not cited yet.