# Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory

arXiv: https://arxiv.org/abs/1910.00359 · HTML: https://mlsanity.com/paper/1910.00359/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 21 of 21)

- [Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models](https://mlsanity.com/paper/2110.13057/) (2021)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [Stochastic Training is Not Necessary for Generalization](https://mlsanity.com/paper/2109.14119/) (2021)
- [A linearized framework and a new benchmark for model selection for fine-tuning](https://mlsanity.com/paper/2102.00084/) (2021)
- [TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels](https://mlsanity.com/paper/2207.06343/) (2022)
- [Winning the Lottery Ahead of Time: Efficient Early Network Pruning](https://mlsanity.com/paper/2206.10451/) (2022)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape](https://mlsanity.com/paper/2305.19510/) (2023)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [Koopman-based generalization bound: New aspect for full-rank weights](https://mlsanity.com/paper/2302.05825/) (2023)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [Understanding Deep Learning via Decision Boundary](https://mlsanity.com/paper/2206.01515/) (2022)
- [On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective](https://mlsanity.com/paper/2310.00137/) (2023)
- [Just How Flexible are Neural Networks in Practice?](https://mlsanity.com/paper/2406.11463/) (2024)
- [Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization](https://mlsanity.com/paper/2505.22578/) (2025)
- [Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSs](https://mlsanity.com/paper/2509.21895/) (2025)
- [Geometric Regularization from Overparameterization](https://mlsanity.com/paper/2202.09276/) (2022)
- [Dynamical loss functions shape landscape topography and improve learning in artificial neural networks](https://mlsanity.com/paper/2410.10690/) (2024)
- [SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks](https://mlsanity.com/paper/2505.15628/) (2025)
- [Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix](https://mlsanity.com/paper/2609.31983/) (2026)

## Abstract (arXiv)

We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role; (4) find that rank does not correlate with generalization or robustness in a practical setting.