mlsanityData to 14 Oct 2026

Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory

arXiv 1910.00359 ↗First posted 1 Oct 2019cs.LG math.OC stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role; (4) find that rank does not correlate with generalization or robustness in a practical setting.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 6; 2023: 3; 2024: 2; 2025: 3; 2026: 1.

21 papers.

Sort:

  1. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 108
  2. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  3. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  4. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  5. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  6. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  7. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  8. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 17
  9. 2021When Are Solutions Connected in Deep Networks?cited by 11
  10. 2023Koopman-based generalization bound: New aspect for full-rank weightscited by 9
  11. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  12. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
9 more
  1. 2022Understanding Deep Learning via Decision Boundarycited by 7
  2. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  3. 2024Just How Flexible are Neural Networks in Practice?cited by 3
  4. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 1
  5. 2025Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSscited by 1
  6. 2022Geometric Regularization from Overparameterizationcited by 0
  7. 2024Dynamical loss functions shape landscape topography and improve learning in artificial neural networkscited by 0
  8. 2025SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Taskscited by 0
  9. 2026Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrixcited by 0