Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networks
Not on the map. Why?
Abstract (arXiv)
We observe a novel `multiple-descent' phenomenon during the learning process of a recurrent neural network called long-short-term memory (LSTM) networks during its training on real-world task, in which the performance goes through long cycles of up and down trends multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in performance -- indicated by loss function in test data -- are closely associated with the phase transition process between order and chaos of the model, and the local optimal training step are consistently at the critical transition point between the two phases. More importantly, the most optimal point of the model usually occurs at the first transition from order to chaos, where the `width' of the `edge of chaos' is often the widest, allowing the best exploration of weight configurations for learning.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 26 May 2025.
Builds on
22 of 43 references.
References · 22
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2019On the Convergence of Adam and Beyondcited by 841
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
- 2015Cyclical Learning Rates for Training Neural Networkscited by 629
- 2017Exploring Generalization in Deep Learningcited by 600
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
10 more
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
- 2018How Does Batch Normalization Help Optimization?cited by 357
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2019Scaling description of generalization with number of parameters in deep learningcited by 83
- 2019Overfitting Mechanism and Avoidance in Deep Neural Networkscited by 30
- 2022Multiple Descent in the Multiple Random Feature Modelcited by 10
- 2019Optimal Machine Intelligence at the Edge of Chaoscited by 7
- 2024Unveiling Multiple Descents in Unsupervised Autoencoderscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.