mlsanityData to 7 Oct 2026

Learning One-hidden-layer Neural Networks with Landscape Design

arXiv 1711.00501 ↗First posted 1 Nov 2017cs.LG cs.DS math.OCcited by 79

Not on the map. Why?

Abstract (arXiv)

We consider the problem of learning a one-hidden-layer neural network: we assume the input x∈ ℝd is from Gaussian distribution and the label y = a^top σ(Bx) + xi, where a is a nonnegative vector in ℝm with m≤ d, B∈ ℝm× d is a full-rank weight matrix, and xi is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously. Inspired by the formula, we design a non-convex objective function G(·) whose landscape is guaranteed to have the following properties: 1. All local minima of G are also global minima. 2. All global minima of G correspond to the ground truth parameters. 3. The value and gradient of G can be estimated using samples. With these properties, stochastic gradient descent on G provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity result and validate the results by simulations.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 27; 2022: 16; 2023: 18; 2024: 10; 2025: 5; 2026: 1.

79 papers.

Sort:

By area:

  1. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  2. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  3. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  4. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 82
  5. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  6. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  7. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  8. 2021Constrained Learning with Non-Convex Lossescited by 42
  9. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  10. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  11. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  12. 2022Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networkscited by 27
38 more
  1. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  2. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  3. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  4. 2024Provably learning a multi-head attention layercited by 21
  5. 2019On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulatorcited by 20
  6. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  7. 2021On the Cryptographic Hardness of Learning Single Periodic Neuronscited by 20
  8. 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
  9. 2021GradSign: Model Performance Inference with Theoretical Insightscited by 18
  10. 2022How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysiscited by 18
  11. 2023On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Explorationcited by 17
  12. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  13. 2021Optimal Gradient-based Algorithms for Non-concave Bandit Optimizationcited by 15
  14. 2021Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscited by 15
  15. 2022Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysiscited by 15
  16. 2023Resilient Constrained Learningcited by 15
  17. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  18. 2022AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inferencecited by 14
  19. 2023Learning Narrow One-Hidden-Layer ReLU Networkscited by 14
  20. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithmscited by 13
  21. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  22. 2022Local Identifiability of Deep ReLU Neural Networks: the Theorycited by 12
  23. 2022Agnostic Learning of General ReLU Activation Using Gradient Descentcited by 12
  24. 2021Escape saddle points by a simple gradient-descent based algorithmcited by 11
  25. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  26. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
  27. 2023Computational Complexity of Learning Neural Networks: Smoothness and Degeneracycited by 10
  28. 2021Going Beyond Linear RL: Sample Efficient Neural Function Approximationcited by 9
  29. 2022Annihilation of Spurious Minima in Two-Layer ReLU Networkscited by 9
  30. 2024Near-Optimal Solutions of Constrained Learning Problemscited by 9
  31. 2021Why Do Local Methods Solve Nonconvex Problems?cited by 8
  32. 2021Scaffolding Setscited by 8
  33. 2023Implicit Regularization for Group Sparsitycited by 8
  34. 2023Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network?cited by 8
  35. 2025Robust Learning of Multi-index Models via Iterative Subspace Approximationcited by 8
  36. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  37. 2023A faster and simpler algorithm for learning shallow networkscited by 7
  38. 2021On the Provable Generalization of Recurrent Neural Networkscited by 6

50 most cited of 79 All 79 (JSON)