mlsanityData to 14 Oct 2026

Deep linear neural networks with arbitrary loss: All local minima are global

arXiv 1712.01473 ↗First posted 5 Dec 2017cs.LG stat.MLcited by 65

Not on the map. Why?

Abstract (arXiv)

We consider deep linear networks with arbitrary convex differentiable loss. We provide a short and elementary proof of the fact that all local minima are global minima if the hidden layers are either 1) at least as wide as the input layer, or 2) at least as wide as the output layer. This result is the strongest possible in the following sense: If the loss is convex and Lipschitz but not differentiable then deep linear networks can have sub-optimal local minima.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 16; 2023: 11; 2024: 5; 2025: 9; 2026: 9.

65 papers.

Sort:

By area:

  1. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  3. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  4. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  5. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  6. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  7. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  8. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  9. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  10. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  11. 2021Geometry of Linear Convolutional Networkscited by 27
  12. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
38 more
  1. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  2. 2022Exact Solutions of a Deep Linear Networkcited by 24
  3. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  4. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  5. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  6. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  7. 2023Regularization properties of adversarially-trained linear regressioncited by 18
  8. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  9. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  10. 2021When Are Solutions Connected in Deep Networks?cited by 11
  11. 2021Understanding Dynamics of Nonlinear Representation Learning and Its Applicationcited by 11
  12. 2022Non-convex online learning via algorithmic equivalencecited by 11
  13. 2024Algebraic Complexity and Neurovariety of Linear Convolutional Networkscited by 10
  14. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  15. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8
  16. 2023Learning Rich Rankingscited by 8
  17. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  18. 2023Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Losscited by 7
  19. 2023Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoderscited by 7
  20. 2022Exact Phase Transitions in Deep Learningcited by 6
  21. 2023Feature Collapsecited by 6
  22. 2022A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networkscited by 4
  23. 2021Theoretical Exploration of Flexible Transmitter Modelcited by 3
  24. 2022Stochastic Neural Networks with Infinite Width are Deterministiccited by 3
  25. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
  26. 2022Learning to Optimize Quasi-Newton Methodscited by 3
  27. 2022A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizescited by 3
  28. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  29. 2021The Landscape of Multi-Layer Linear Neural Network From the Perspective of Algebraic Geometrycited by 2
  30. 2022Memorization-Dilation: Modeling Neural Collapse Under Label Noisecited by 2
  31. 2025Parameter Expanded Stochastic Gradient Markov Chain Monte Carlocited by 2
  32. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 2
  33. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  34. 2023Enhancing Adversarial Attacks: The Similar Target Methodcited by 1
  35. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  36. 2024Algebraic Representations for Faster Predictions in Convolutional Neural Networkscited by 1
  37. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  38. 2025Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networkscited by 1

50 most cited of 65 All 65 (JSON)