mlsanityData to 7 Oct 2026

Depth Creates No Bad Local Minima

arXiv 1702.08580 ↗First posted 27 Feb 2017cs.LG cs.NE math.OCcited by 29

Not on the map. Why?

Abstract (arXiv)

In deep learning, textitdepth, as well as textitnonlinearity, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep linear neural networks are global minima. Our theoretical results generalize previous results with fewer assumptions, and this analysis provides a method to show similar results beyond square loss in deep linear models.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 4; 2024: 5; 2025: 3; 2026: 1.

29 papers.

Sort:

By area:

  1. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  2. 2021Geometry of Linear Convolutional Networkscited by 27
  3. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  4. 2022Exact Solutions of a Deep Linear Networkcited by 24
  5. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  6. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  7. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  8. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  9. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  10. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  11. 2021When Are Solutions Connected in Deep Networks?cited by 10
  12. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
17 more
  1. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  2. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  3. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8
  4. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  5. 2022Understanding Deep Learning via Decision Boundarycited by 7
  6. 2023Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Losscited by 7
  7. 2022DiffML: End-to-end Differentiable ML Pipelinescited by 6
  8. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
  9. 2022A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networkscited by 3
  10. 2021The Landscape of Multi-Layer Linear Neural Network From the Perspective of Algebraic Geometrycited by 1
  11. 2022Learning Neural Ranking Models Online from Implicit User Feedbackcited by 1
  12. 2024Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Inputcited by 1
  13. 2025System Identification and Control Using Lyapunov-Based Deep Neural Networks without Persistent Excitation: A Concurrent Learning Approachcited by 1
  14. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 1
  15. 2021Neural Networks with Complex-Valued Weights Have No Spurious Local Minimacited by 0
  16. 2022Global Convergence Analysis of Deep Linear Networks with A One-neuron Layercited by 0
  17. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0