mlsanityData to 7 Oct 2026

Qualitatively characterizing neural network optimization problems

arXiv 1412.6544 ↗First posted 19 Dec 2014cs.NE cs.LG stat.MLcited by 186

Not on the map. Why?

Abstract (arXiv)

Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 36; 2023: 35; 2024: 35; 2025: 24; 2026: 24.

186 papers.

Sort:

By area:

  1. 2021Robust fine-tuning of zero-shot modelscited by 785
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  4. 2022Fusing finetuned models for better pretrainingcited by 110
  5. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  6. 2023Deep Model Fusion: A Surveycited by 89
  7. 2022When Do Flat Minima Optimizers Work?cited by 78
  8. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  9. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  10. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  11. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  12. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
38 more
  1. 2022Self-Ensemble Adversarial Training for Improved Robustnesscited by 38
  2. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  3. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  4. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  5. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  6. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  7. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  8. 2021AdvRush: Searching for Adversarially Robust Neural Architecturescited by 28
  9. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  10. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  11. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  12. 2021Selfish Sparse RNN Trainingcited by 26
  13. 2021Finding Discriminative Filters for Specific Degradations in Blind Super-Resolutioncited by 26
  14. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  15. 2022Convergence of gradient descent for deep neural networkscited by 24
  16. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  17. 2022Random initialisations performing above chance and how to find themcited by 23
  18. 2022Random matrix analysis of deep neural network weight matricescited by 21
  19. 2023Sparse is Enough in Fine-tuning Pre-trained Large Language Modelscited by 21
  20. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  21. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  22. 2021GradSign: Model Performance Inference with Theoretical Insightscited by 18
  23. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  24. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  25. 2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
  26. 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16
  27. 2021Robust Optimization for Multilingual Translation with Imbalanced Datacited by 15
  28. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  29. 2021Exploring the Geometry and Topology of Neural Network Loss Landscapescited by 14
  30. 2021Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curationcited by 13
  31. 2024What is Dataset Distillation Learning?cited by 13
  32. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  33. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 13
  34. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  35. 2024Data-Driven Physics-Informed Neural Networks: A Digital Twin Perspectivecited by 12
  36. 2023Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Controlcited by 11
  37. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  38. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10

50 most cited of 186 All 186 (JSON)