mlsanityData to 14 Oct 2026

Optimal Activation Functions for the Random Features Regression Model

arXiv 2206.01332 ↗First posted 31 May 2022stat.ML cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The asymptotic mean squared test error and sensitivity of the Random Features Regression model (RFR) have been recently studied. We build on this work and identify in closed-form the family of Activation Functions (AFs) that minimize a combination of the test error and sensitivity of the RFR under different notions of functional parsimony. We find scenarios under which the optimal AFs are linear, saturated linear functions, or expressible in terms of Hermite polynomials. Finally, we show how using optimal AFs impacts well-established properties of the RFR model, such as its double descent curve, and the dependency of its optimal regularization parameter on the observation noise level.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 44 of its 80 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

44 of 80 references.

References · 44

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  6. 2018Robustness May Be at Odds with Accuracycited by 880
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  9. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  10. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  11. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  12. 2019Two models of double descent for weak featurescited by 195
32 more
  1. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  2. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  3. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  4. 2019Linearized two-layers neural networks in high dimensioncited by 155
  5. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  6. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138
  7. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  8. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  9. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  10. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  11. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  12. 2018Does data interpolation contradict statistical optimality?cited by 95
  13. 2017Regularization for Deep Learning: A Taxonomycited by 91
  14. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  15. 2022Universality of empirical risk minimizationcited by 75
  16. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  17. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 63
  18. 2020A law of robustness for two-layers neural networkscited by 55
  19. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  20. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 45
  21. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 41
  22. 2018A representer theorem for deep neural networkscited by 38
  23. 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
  24. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  25. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  26. 2020SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustnesscited by 18
  27. 2020Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensionscited by 13
  28. 2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13
  29. 2018PLU: The Piecewise Linear Unit Activation Functioncited by 11
  30. 2020Soft-Root-Sign Activation Functioncited by 8
  31. 2021Learning specialized activation functions with the Piecewise Linear Unitcited by 7
  32. 2019Improved Adversarial Robustness by Reducing Open Space Risk via Tent Activationscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2.

4 papers.

  1. 2023Optimal Nonlinearities Improve Generalization Performance of Random Featurescited by 3
  2. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  3. 2024Learning Expressive Random Feature Models via Parametrized Activationscited by 1
  4. 2023Understanding the Role of Optimization in Double Descentcited by 0