mlsanityData to 14 Oct 2026

Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization

arXiv 2406.18035 ↗First posted 26 Jun 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Determining whether deep neural network (DNN) models can reliably recover target functions at overparameterization is a critical yet complex issue in the theory of deep learning. To advance understanding in this area, we introduce a concept we term "local linear recovery" (LLR), a weaker form of target function recovery that renders the problem more amenable to theoretical analysis. In the sense of LLR, we prove that functions expressible by narrower DNNs are guaranteed to be recoverable from fewer samples than model parameters. Specifically, we establish upper limits on the optimistic sample sizes, defined as the smallest sample size necessary to guarantee LLR, for functions in the space of a given DNN. Furthermore, we prove that these upper bounds are achieved in the case of two-layer tanh neural networks. Our research lays a solid groundwork for future investigations into the recovery capabilities of DNNs in overparameterized scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

18 of 25 references.

References · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  5. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  6. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  7. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  8. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  9. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  10. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  11. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  12. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
6 more
  1. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 30
  2. 2022Implicit regularization of dropoutcited by 25
  3. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
  4. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  5. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  6. 2023Optimistic Estimate Uncovers the Potential of Nonlinear Modelscited by 5

Built on by

Not cited yet.