mlsanityData to 7 Oct 2026

Towards Understanding the Optimization Mechanisms in Deep Learning

arXiv 2503.23016 ↗First posted 29 Mar 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model's parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 72 references that are in the map. First posted on arXiv 29 Mar 2025.

Builds on

47 of 72 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  12. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
35 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2021Going deeper with Image Transformerscited by 759
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2017Deep Neural Networks as Gaussian Processescited by 617
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  12. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  13. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  14. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  15. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  16. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  17. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  18. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  19. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  20. 2019Learning with Fenchel-Young Lossescited by 117
  21. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  22. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  23. 2019Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programmingcited by 100
  24. 2025Transformers without Normalizationcited by 69
  25. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  26. 2019Scaleable input gradient regularization for adversarial robustnesscited by 45
  27. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  28. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  29. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  30. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  31. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  32. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  33. 2022Annihilation of Spurious Minima in Two-Layer ReLU Networkscited by 9
  34. 2024XAIport: A Service Framework for the Early Adoption of XAI in AI Model Developmentcited by 3
  35. 2024An Open API Architecture to Discover the Trustworthy Explanation of Cloud AI Servicescited by 2

Built on by

Not cited yet.