# Semi-flat minima and saddle points by embedding neural networks to overparameterization

arXiv: https://arxiv.org/abs/1906.04868 · HTML: https://mlsanity.com/paper/1906.04868/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 30)

- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [Embedding Principle of Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2105.14573/) (2021)
- [Towards Understanding the Condensation of Neural Networks at Initial Training](https://mlsanity.com/paper/2105.11686/) (2021)
- [Embedding Principle: a hierarchical structure of loss landscape of deep neural networks](https://mlsanity.com/paper/2111.15527/) (2021)
- [An overview of condensation phenomenon in deep learning](https://mlsanity.com/paper/2504.09484/) (2025)
- [Loss Spike in Training Neural Networks](https://mlsanity.com/paper/2305.12133/) (2023)
- [Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures](https://mlsanity.com/paper/2512.20607/) (2025)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity](https://mlsanity.com/paper/2404.06391/) (2024)
- [Understanding the Initial Condensation of Convolutional Neural Networks](https://mlsanity.com/paper/2305.09947/) (2023)
- [Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks](https://mlsanity.com/paper/2205.13283/) (2022)
- [Linear Stability Hypothesis and Rank Stratification for Nonlinear Models](https://mlsanity.com/paper/2211.11623/) (2022)
- [Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks](https://mlsanity.com/paper/2305.05089/) (2023)
- [Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding](https://mlsanity.com/paper/2402.05626/) (2024)
- [Optimistic Estimate Uncovers the Potential of Nonlinear Models](https://mlsanity.com/paper/2307.08921/) (2023)
- [Towards Understanding Adam Convergence on Highly Degenerate Polynomials](https://mlsanity.com/paper/2603.09581/) (2026)
- [Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion](https://mlsanity.com/paper/2405.13721/) (2024)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks](https://mlsanity.com/paper/2405.17501/) (2024)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)
- [Stimulus symmetries can confound representational similarity analyses](https://mlsanity.com/paper/2605.21324/) (2026)
- [Proximity to Losslessly Compressible Parameters](https://mlsanity.com/paper/2306.02834/) (2023)
- [Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization](https://mlsanity.com/paper/2406.18035/) (2024)
- [Neural Network-based High-index Saddle Dynamics Method for Searching Saddle Points and Solution Landscape](https://mlsanity.com/paper/2411.16200/) (2024)
- [Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Lifting](https://mlsanity.com/paper/2505.13582/) (2025)
- [Zero Generalization Error Theorem for Random Interpolators via Algebraic Geometry](https://mlsanity.com/paper/2512.06347/) (2025)
- [Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario](https://mlsanity.com/paper/2604.02393/) (2026)
- [A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks](https://mlsanity.com/paper/2606.04327/) (2026)
- [Explaining Machine Learning and Memorization with Statistical Mechanics](https://mlsanity.com/paper/2606.31110/) (2026)
- [Removing spurious minima for planar features by skip connections](https://mlsanity.com/paper/2610.01728/) (2026)

## Abstract (arXiv)

We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our results show that the networks with smooth and ReLU activation have different partially flat landscapes around the embedded point. We also relate these results to a difference of their generalization abilities in overparameterized realization.