mlsanityData to 7 Oct 2026

Iterative regularization in classification via hinge loss diagonal descent

arXiv 2212.12675 ↗First posted 24 Dec 2022stat.ML cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Iterative regularization is a classic idea in regularization theory, that has recently become popular in machine learning. On the one hand, it allows to design efficient algorithms controlling at the same time numerical and statistical accuracy. On the other hand it allows to shed light on the learning curves observed while training neural networks. In this paper, we focus on iterative regularization in the context of classification. After contrasting this setting with that of linear inverse problems, we develop an iterative regularization approach based on the use of the hinge loss function. More precisely we consider a diagonal approach for a family of algorithms for which we prove convergence as well as rates of convergence and stability results for a suitable classification noise model. Our approach compares favorably with other alternatives, as confirmed by numerical simulations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 103 references that are in the map. First posted on arXiv 24 Dec 2022.

Builds on

35 of 103 references.

References · 35

  1. 2021Convex optimizationcited by 2,479
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  4. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  12. 2018Convergence of Gradient Descent on Separable Datacited by 108
23 more
  1. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  2. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 84
  3. 2020Gradient descent follows the regularization path for general lossescited by 57
  4. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  5. 2015From error bounds to the complexity of first-order descent methods for convex functionscited by 42
  6. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  7. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  8. 2019Beating SGD Saturation with Tail-Averaging and Minibatchingcited by 21
  9. 2020Mirrorless Mirror Descent: A Natural Derivation of Mirror Descentcited by 18
  10. 2014Convergence of Stochastic Proximal Gradient Algorithmcited by 17
  11. 2019Implicit Regularization of Accelerated Methods in Hilbert Spacescited by 14
  12. 2015Linearly Convergent Away-Step Conditional Gradient for Non-strongly Convex Functionscited by 12
  13. 2021Convergence rates for the Heavy-Ball continuous dynamics for non-convex optimization, under Polyak-L ojasiewicz conditioncited by 11
  14. 2015Iterative Regularization for Learning with Convex Loss Functionscited by 9
  15. 2017Convergence of the Forward-Backward Algorithm: Beyond the Worst Case with the Help of Geometrycited by 7
  16. 2021Ada-BKB: Scalable Gaussian Process Optimization on Continuous Domains by Adaptive Discretizationcited by 7
  17. 2014Towards A Deeper Geometric, Analytic and Algorithmic Understanding of Marginscited by 4
  18. 2017Don't relax: early stopping for convex regularizationcited by 4
  19. 2021From inexact optimization to learning via gradient concentrationcited by 4
  20. 2016Iterative regularization via dual diagonal descentcited by 3
  21. 2019Bias of Homotopic Gradient Descent for the Hinge Losscited by 3
  22. 2019Accelerated iterative regularization via dual diagonal descentcited by 3
  23. 2022Stochastic mirror descent method for linear ill-posed problems in Banach spacescited by 2

Built on by

Not cited yet.