mlsanityData to 14 Oct 2026

Implicit regularisation in stochastic gradient descent: from single-objective to two-player games

arXiv 2307.05789 ↗First posted 11 Jul 2023stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent years have seen many insights on deep learning optimisation being brought forward by finding implicit regularisation effects of commonly used gradient-based optimisers. Understanding implicit regularisation can not only shed light on optimisation dynamics, but it can also be used to improve performance and stability across problem domains, from supervised learning to two-player games such as Generative Adversarial Networks. An avenue for finding such implicit regularisation effects has been quantifying the discretisation errors of discrete optimisers via continuous-time flows constructed by backward error analysis (BEA). The current usage of BEA is not without limitations, since not all the vector fields of continuous-time flows obtained using BEA can be written as a gradient, hindering the construction of modified losses revealing implicit regularisers. In this work, we provide a novel approach to use BEA, and show how our approach can be used to construct continuous-time flows with vector fields that can be written as gradients. We then use this to find previously unknown implicit regularisation effects, such as those induced by multiple stochastic gradient descent steps while accounting for the exact data batches used in the updates, and in generally differentiable two-player games.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 19 of its 20 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

19 of 20 references.

References · 19

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  5. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  10. 2020Implicit Gradient Regularizationcited by 146
  11. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  12. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
7 more
  1. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  2. 2022A Review of Deep Transfer Learning and Recent Advancementscited by 57
  3. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  4. 2020Uncertainty Estimation Using a Single Deep Deterministic Neural Networkcited by 37
  5. 2019Implicit competitive regularization in GANscited by 14
  6. 2021Discretization Drift in Two-Player Gamescited by 14
  7. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  2. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0