mlsanityData to 7 Oct 2026

Semi-flat minima and saddle points by embedding neural networks to overparameterization

arXiv 1906.04868 ↗First posted 12 Jun 2019cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our results show that the networks with smooth and ReLU activation have different partially flat landscapes around the embedded point. We also relate these results to a difference of their generalization abilities in overparameterized realization.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 2; 2023: 5; 2024: 6; 2025: 6; 2026: 5.

29 papers.

Sort:

  1. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  2. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  3. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  4. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  5. 2023Loss Spike in Training Neural Networkscited by 15
  6. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  7. 2025An overview of condensation phenomenon in deep learningcited by 13
  8. 2021When Are Solutions Connected in Deep Networks?cited by 10
  9. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  10. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  11. 2022Linear Stability Hypothesis and Rank Stratification for Nonlinear Modelscited by 7
  12. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 7
17 more
  1. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  2. 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 5
  3. 2023Optimistic Estimate Uncovers the Potential of Nonlinear Modelscited by 5
  4. 2024Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completioncited by 3
  5. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  6. 2026Towards Understanding Adam Convergence on Highly Degenerate Polynomialscited by 3
  7. 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
  8. 2025Saddle Hierarchy in Dense Associative Memorycited by 1
  9. 2026Stimulus symmetries can confound representational similarity analysescited by 1
  10. 2023Proximity to Losslessly Compressible Parameterscited by 0
  11. 2024Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterizationcited by 0
  12. 2024Neural Network-based High-index Saddle Dynamics Method for Searching Saddle Points and Solution Landscapecited by 0
  13. 2025Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Liftingcited by 0
  14. 2025Zero Generalization Error Theorem for Random Interpolators via Algebraic Geometrycited by 0
  15. 2026Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenariocited by 0
  16. 2026A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networkscited by 0
  17. 2026Removing spurious minima for planar features by skip connectionscited by 0