Flat Channels to Infinity in Neural Loss Landscapes
Not on the map. Why?
Abstract (arXiv)
The loss landscapes of neural networks contain minima and saddle points that may be connected in flat regions or appear in isolation. We identify and characterize a special structure in the loss landscape: channels along which the loss decreases extremely slowly, while the output weights of at least two neurons, ai and aj, diverge to ±infinity, and their input weight vectors, wi and wj, become equal to each other. At convergence, the two neurons implement a gated linear unit: aiσ(wi · x) + ajσ(wj · x) rightarrow σ(w · x) + (v · x) σ'(w · x). Geometrically, these channels to infinity are asymptotically parallel to symmetry-induced lines of critical points. Gradient flow solvers, and related optimization methods like SGD or ADAM, reach the channels with high probability in diverse regression settings, but without careful inspection they look like flat local minima with finite parameter values. Our characterization provides a comprehensive picture of these quasi-flat regions in terms of gradient dynamics, geometry, and functional interpretation. The emergence of gated linear units at the end of the channels highlights a surprising aspect of the computational capabilities of fully connected layers.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 17 Jun 2025.
Builds on
48 of 62 references.
References · 48
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2016Language Modeling with Gated Convolutional Networkscited by 702
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
- 2020Federated Learning with Matched Averagingcited by 564
- 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
36 more
- 2017How to Escape Saddle Points Efficientlycited by 285
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
- 2019Model Fusion via Optimal Transportcited by 219
- 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
- 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 132
- 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
- 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
- 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
- 2017The loss surface of deep and wide neural networkscited by 74
- 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
- 2020The Global Landscape of Neural Networks: An Overviewcited by 57
- 2019Splitting Steepest Descent for Growing Neural Architecturescited by 55
- 2018The loss landscape of overparameterized neural networkscited by 39
- 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
- 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
- 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
- 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 30
- 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 30
- 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
- 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
- 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
- 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
- 2023Expand-and-Cluster: Parameter Recovery of Neural Networkscited by 15
- 2020Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splittingcited by 14
- 2022Decomposing neural networks as mappings of correlation functionscited by 12
- 2019Elimination of All Bad Local Minima in Deep Learningcited by 10
- 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
- 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 9
- 2018Non-attracting Regions of Local Minima in Deep and Wide Neural Networkscited by 8
- 2019Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinitycited by 6
- 2024Do Deep Neural Network Solutions Form a Star Domain?cited by 6
- 2019Best k-layer neural network approximationscited by 4
- 2023MLPGradientFlow: going with the flow of multilayer perceptrons (and finding minima fast and accurately)cited by 3
- 2023Certifying the absence of spurious local minima at infinitycited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.