mlsanityData to 7 Oct 2026

Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"

arXiv 2012.06898 ↗First posted 12 Dec 2020cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

We revisit and extend the experiments of Goodfellow et al. (2014), who showed that - for then state-of-the-art networks - "the objective function has a simple, approximately convex shape" along the linear path between initialization and the trained weights. We do not find this to be the case for modern networks on CIFAR-10 and ImageNet. Instead, although loss is roughly monotonically non-increasing along this path, it remains high until close to the optimum. In addition, training quickly becomes linearly separated from the optimum by loss barriers. We conclude that, although Goodfellow et al.'s findings describe the "relatively easy to optimize" MNIST setting, behavior is qualitatively different in modern settings.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 4; 2024: 4; 2025: 1; 2026: 2.

22 papers.

Sort:

  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2022Fusing finetuned models for better pretrainingcited by 110
  3. 2023Deep Model Fusion: A Surveycited by 89
  4. 2021Learning Neural Network Subspacescited by 82
  5. 2022When Do Flat Minima Optimizers Work?cited by 78
  6. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  7. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  8. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  9. 2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
  10. 2021Connecting Low-Loss Subspace for Personalized Federated Learningcited by 12
  11. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10
  12. 2021Personalized Algorithm Generation: A Case Study in Learning ODE Integratorscited by 8
10 more
  1. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  2. 2023High-dimensional manifold of solutions in neural networks: insights from statistical physicscited by 4
  3. 2022FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networkscited by 2
  4. 2023Transferring Learning Trajectories of Neural Networkscited by 2
  5. 2024Weight Scope Alignment: A Frustratingly Easy Method for Model Mergingcited by 2
  6. 2024Exploring and Exploiting the Asymmetric Valley of Deep Neural Networkscited by 1
  7. 2024Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networkscited by 1
  8. 2022On the Subspace Structure of Gradient-Based Meta-Learningcited by 0
  9. 2026Neural Network Quantization by Learning Low-Loss Subspacescited by 0
  10. 2026Scalar Representations of Neural Network Training Dynamicscited by 0