Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy
Not on the map. Why?
Abstract (arXiv)
We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich directional structure of optimization trajectories, represented by their pointwise parameters. Towards this end, we introduce some natural notions of the complexity of optimization trajectories, both qualitative and quantitative, which hallmark the directional nature of optimization in neural networks: when is there redundancy, and when exploration. We use them to reveal the inherent nuance and interplay involved between various optimization choices, such as momentum and weight decay. Further, the trajectory perspective helps us see the effect of scale on regularizing the directional nature of trajectories, and as a by-product, we also observe an intriguing heterogeneity of Q,K,V dynamics in the middle attention layers in LLMs and which is homogenized by scale. Importantly, we put the significant directional redundancy observed to the test by demonstrating that training only scalar batchnorm parameters some while into training matches the performance of training the entire network, which thus exhibits the potential of hybrid optimization schemes that are geared towards efficiency.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 12 Mar 2024.
Builds on
25 of 35 references.
References · 25
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2017Decoupled Weight Decay Regularizationcited by 1,722
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2023Studying Large Language Model Generalization with Influence Functionscited by 311
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
- 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
13 more
- 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
- 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
- 2020Relative Flatness and Generalizationcited by 88
- 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
- 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
- 2020Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descentcited by 40
- 2022Towards understanding how momentum improves generalization in deep learningcited by 40
- 2021SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Searchcited by 35
- 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
- 2021Why flatness does and does not correlate with generalization for deep neural networkscited by 13
- 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
- 2017Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activationscited by 8
- 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.