Dynamical loss functions shape landscape topography and improve learning in artificial neural networks
Not on the map. Why?
Abstract (arXiv)
Dynamical loss functions are derived from standard loss functions used in supervised classification tasks, but are modified so that the contribution from each class periodically increases and decreases. These oscillations globally alter the loss landscape without affecting the global minima. In this paper, we demonstrate how to transform cross-entropy and mean squared error into dynamical loss functions. We begin by discussing the impact of increasing the size of the neural network or the learning rate on the depth and sharpness of the minima that the system explores. Building on this intuition, we propose several versions of dynamical loss functions and use a simple classification problem where we can show how they significantly improve validation accuracy for networks of varying sizes. Finally, we explore how the landscape of these dynamical loss functions evolves during training, highlighting the emergence of instabilities that may be linked to edge-of-instability minimization.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 63 references that are in the map. First posted on arXiv 14 Oct 2024.
Builds on
31 of 63 references.
References · 31
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
- 2014The Loss Surfaces of Multilayer Networkscited by 278
- 2018Gradient Descent Happens in a Tiny Subspacecited by 208
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
19 more
- 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
- 2022Understanding the unstable convergence of gradient descentcited by 84
- 2019Scaling description of generalization with number of parameters in deep learningcited by 83
- 2018A Walk with SGDcited by 74
- 2017On Loss Functions for Deep Neural Networks in Classificationcited by 68
- 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
- 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
- 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
- 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
- 2018The loss landscape of overparameterized neural networkscited by 38
- 2017The landscape of the spiked tensor modelcited by 34
- 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
- 2020Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized modelscited by 26
- 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
- 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
- 2020Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Trainingcited by 11
- 2023The star-shaped space of solutions of the spherical negative perceptroncited by 8
- 2021Tilting the playing field: Dynamical loss functions for machine learningcited by 7
- 2021Model architecture can transform catastrophic forgetting into positive transfercited by 3
Built on by
Not cited yet.