mlsanityData to 14 Oct 2026

Learning a Single Neuron with Bias Using Gradient Descent

arXiv 2106.01101 ↗First posted 2 Jun 2021cs.LG cs.NE stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

We theoretically study the fundamental problem of learning a single neuron with a bias term (x mapsto σ(<w,x> + b)) in the realizable setting with the ReLU activation, using gradient descent. Perhaps surprisingly, we show that this is a significantly different and more challenging problem than the bias-less case (which was the focus of previous works on single neurons), both in terms of the optimization geometry as well as the ability of gradient methods to succeed in some scenarios. We provide a detailed study of this problem, characterizing the critical points of the objective, demonstrating failure cases, and providing positive convergence guarantees under different sets of assumptions. To prove our results, we develop some tools which may be of independent interest, and improve previous results on learning single neurons.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

21 of 27 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014Convex Optimization: Algorithms and Complexitycited by 575
  3. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  4. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 88
  5. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  6. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  7. 2017Learning ReLUs via Gradient Descentcited by 54
  8. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  9. 2020Learning a Single Neuron with Gradient Methodscited by 50
  10. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  11. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  12. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
9 more
  1. 2016Reliably Learning the ReLU in Polynomial Timecited by 36
  2. 2020Tight Hardness Results for Training Depth-2 ReLU Networkscited by 35
  3. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  4. 2020Approximation Schemes for ReLU Regressioncited by 34
  5. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  6. 2020Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexitycited by 25
  7. 2017When is a Convolutional Filter Easy To Learn?cited by 21
  8. 2014A Stochastic PCA and SVD Algorithm with an Exponential Convergence Ratecited by 18
  9. 2019Fitting ReLUs via SGD and Quantized SGDcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 4; 2025: 1; 2026: 4.

17 papers.

Sort:

  1. 2021Optimization-Based Separations for Neural Networkscited by 17
  2. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  3. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  4. 2022Agnostic Learning of General ReLU Activation Using Gradient Descentcited by 12
  5. 2024Self-Normalized Resets for Plasticity in Continual Learningcited by 11
  6. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  7. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  8. 2023Max-affine regression via first-order methodscited by 3
  9. 2025Pay Attention to Small Weightscited by 3
  10. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  11. 2026Deep Networks Learn Deep Hierarchical Modelscited by 2
  12. 2024RedEx: Beyond Fixed Representation Methods via Convex Optimizationcited by 1
5 more
  1. 2026When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networkscited by 1
  2. 2022Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networkscited by 0
  3. 2024Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Targetcited by 0
  4. 2026Polynomial-Time Robust Multiclass Linear Classification under Gaussian Marginalscited by 0
  5. 2026Robust Regression of General ReLUs with Queriescited by 0