Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective
Not on the map. Why?
Abstract (arXiv)
Can we design a model such that its stochastic training favours a desired class of solutions without enforcing an explicit penalty? Under suitable conditions, the interplay between symmetries of a model's weight parametrization and stochastic training favours particular solutions, inducing an implicit bias. Building on this mechanism, we develop a framework for inverse-designing such biases by constructing novel parametrizations and their associated symmetries. We show how holomorphic functions make this construction and calculation simple and explicit. Specifically, we introduce a new parametrization that biases learned weights toward the binary values -1,+1. Numerical experiments confirm the theoretical predictions. They also show that our parametrization reproduces the same preference induced by an explicitly regularized model without adding a penalty to the training loss.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 10 Jan 2026.
Builds on
25 of 34 references.
References · 25
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
- 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
- 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
- 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
- 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
- 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
- 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
- 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
13 more
- 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
- 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
- 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
- 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
- 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
- 2022Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutionscited by 22
- 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
- 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 13
- 2024The Implicit Bias of Gradient Descent on Separable Multiclass Datacited by 11
- 2025Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networkscited by 9
- 2025An entropy formula for the Deep Linear Networkcited by 8
- 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
- 2022Recent Advances in Algebraic Geometry and Bayesian Statisticscited by 4
Built on by
Citing papers per year: 2026: 3.
3 papers.