mlsanityData to 14 Oct 2026

Learning a Single Neuron for Non-monotonic Activation Functions

arXiv 2202.08064 ↗First posted 16 Feb 2022stat.ML cs.LG cs.NEcited by 13

Not on the map. Why?

Abstract (arXiv)

We study the problem of learning a single neuron xmapsto σ(wTx) with gradient descent (GD). All the existing positive results are limited to the case where σ is monotonic. However, it is recently observed that non-monotonic activation functions outperform the traditional monotonic ones in many applications. To fill this gap, we establish learnability without assuming monotonicity. Specifically, when the input distribution is the standard Gaussian, we show that mild conditions on σ (e.g., σ has a dominating linear part) are sufficient to guarantee the learnability in polynomial time and polynomial samples. Moreover, with a stronger assumption on the activation function, the condition of input distribution can be relaxed to a non-degeneracy of the marginal distribution. We remark that our conditions on σ are satisfied by practical non-monotonic activation functions, such as SiLU/Swish and GELU. We also discuss how our positive results are related to existing negative results on training two-layer neural networks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

27 of 41 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  6. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  7. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  8. 2016A Geometric Analysis of Phase Retrievalcited by 92
  9. 2020Smooth Adversarial Trainingcited by 89
  10. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 88
  11. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  12. 2014On the Computational Efficiency of Training Neural Networkscited by 71
15 more
  1. 2018Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrievalcited by 71
  2. 2017Learning ReLUs via Gradient Descentcited by 54
  3. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  4. 2020Learning a Single Neuron with Gradient Methodscited by 50
  5. 2020Algorithms and SQ Lower Bounds for PAC Learning One-Hidden-Layer ReLU Networkscited by 49
  6. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  7. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  8. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  9. 2016Distribution-Specific Hardness of Learning Neural Networkscited by 27
  10. 2018Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitionscited by 23
  11. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  12. 2021Reproducing Activation Function for Deep Learningcited by 19
  13. 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
  14. 2019Fitting ReLUs via SGD and Quantized SGDcited by 10
  15. 2020When Hardness of Approximation Meets Hardness of Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 1; 2025: 5; 2026: 2.

13 papers.

Sort:

  1. 2023On Single Index Models beyond Gaussian Datacited by 17
  2. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  3. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  4. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  5. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  6. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 3
  7. 2023The Local Landscape of Phase Retrieval Under Limited Samplescited by 1
  8. 2025Joint Learning in the Gaussian Single Index Modelcited by 1
  9. 2026Interactive Learning of Single-Index Models via Stochastic Gradient Descentcited by 1
  10. 2024Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Targetcited by 0
  11. 2025AUWave: A Data-Driven Model for Reconstructing Significant Wave Heights Using Sparse Observationscited by 0
  12. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
1 more
  1. 2026Feature Learning for the High Dimensional Stationary Sch"odinger Equation with Deep Ritz Methodcited by 0