mlsanityData to 14 Oct 2026

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?

arXiv 2504.12883 ↗First posted 17 Apr 2025cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understanding their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incorporate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encompasses a broad class of problems, including sparse coding, matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regularization and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

37 of 60 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2021Convex optimizationcited by 2,515
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  7. 2017Implicit Regularization in Matrix Factorizationcited by 322
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  10. 2023Efficient Large Language Models: A Surveycited by 186
  11. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  12. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
25 more
  1. 2017Regularization for Deep Learning: A Taxonomycited by 91
  2. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  3. 2023Transformers as Support Vector Machinescited by 74
  4. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  5. 2016A survey of sparse representation: algorithms and applicationscited by 58
  6. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  7. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  8. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  9. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  10. 2024Weight decay induces low-rank attention layerscited by 27
  11. 2018Hessian Riemannian gradient flows in convex programmingcited by 26
  12. 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 26
  13. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  14. 2020On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspectivecited by 25
  15. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 19
  16. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  17. 2024Mask in the Mirror: Implicit Sparsificationcited by 14
  18. 2015Why Regularized Auto-Encoders learn Sparse Representation?cited by 11
  19. 2021Implicit Regularization in Matrix Sensing via Mirror Descentcited by 11
  20. 2024Implicit Bias of Mirror Flow on Separable Datacited by 9
  21. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  22. 2024Mirror Descent Algorithms with Nearly Dimension-Independent Rates for Differentially-Private Stochastic Saddle-Point Problemscited by 7
  23. 2020Explicit Regularization of Stochastic Gradient Methods through Dualitycited by 6
  24. 2020Linear Convergence of Generalized Mirror Descent with Time-Dependent Mirrorscited by 5
  25. 2022Explicit Regularization via Regularizer Mirror Descentcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  2. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 4
  3. 2025Neural Graduated Assignment for Maximum Common Edge Subgraphscited by 3
  4. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  5. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  6. 2026Does Weight Decay Enhance Training Stability?cited by 1
  7. 2026Policy Regret for Embedding Model Routing: Contextual Bandits with Low-Rank Expertscited by 1
  8. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  9. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0
  10. 2026SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Trainingcited by 0
  11. 2026Mind the Drift: Diagonal Linear Networks Under Large Learning Ratescited by 0
  12. 2026From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muoncited by 0