mlsanityData to 7 Oct 2026

Nonuniform random feature models using derivative information

arXiv 2410.02132 ↗First posted 3 Oct 2024cs.LG cs.NA math.NAcited by 2

Not on the map. Why?

Abstract (arXiv)

We propose nonuniform data-driven parameter distributions for neural network initialization based on derivative data of the function to be approximated. These parameter distributions are developed in the context of non-parametric regression models based on shallow neural networks, and compare favorably to well-established uniform random feature models based on conventional weight initialization. We address the cases of Heaviside and ReLU activation functions, and their smooth approximations (sigmoid and softplus), and use recent results on the harmonic analysis and sparse representation of neural networks resulting from fully trained optimal networks. Extending analytic results that give exact representation, we obtain densities that concentrate in regions of the parameter space corresponding to neurons that are well suited to model the local derivatives of the unknown function. Based on these results, we suggest simplifications of these exact densities based on approximate derivative data in the input points that allow for very efficient sampling and lead to performance of random feature models close to optimal networks in several scenarios.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 70 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

38 of 70 references.

References · 38

  1. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2021Deep learning: a statistical viewpointcited by 239
  6. 2016Generalization Properties of Learning with Random Featurescited by 163
  7. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  8. 2017Sobolev Training for Neural Networkscited by 136
  9. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  10. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  11. 2019How do infinite width bounded norm networks look in function space?cited by 105
  12. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
26 more
  1. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  2. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  3. 2015Neural Network with Unbounded Activation Functions is Universal Approximatorcited by 63
  4. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  5. 2017Bayesian Optimization with Gradientscited by 59
  6. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  7. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  8. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  9. 2019Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equationscited by 40
  10. 2020Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEscited by 40
  11. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 40
  12. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  13. 2018Sparsity of solutions for variational inverse problems with finite-dimensional datacited by 25
  14. 2018On Representer Theorems and Convex Regularizationcited by 24
  15. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  16. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
  17. 2023TransNet: Transferable Neural Networks for Partial Differential Equationscited by 13
  18. 2015The Alternating Descent Conditional Gradient Method for Sparse Inverse Problemscited by 11
  19. 2019Neural network integral representations with the ReLU activation functioncited by 8
  20. 2020Nonconvex regularization for sparse neural networkscited by 7
  21. 2023On the relationship between multivariate splines and infinitely-wide neural networkscited by 5
  22. 2019Linear convergence of accelerated conditional gradient algorithms in spaces of measurescited by 3
  23. 2019On the linear convergence rates of exchange and continuous methods for total variation minimizationcited by 3
  24. 2023On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networkscited by 3
  25. 2013Nonparametric Weight Initialization of Neural Networks via Integral Representationcited by 1
  26. 2023Optimal feedback control of dynamical systems via value-function approximationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Solving Nonlinear PDEs with Sparse Radial Basis Function Networkscited by 3
  2. 2026Sparse RBF Networks for PDEs and nonlocal equations: function space theory, operator calculus, and training algorithmscited by 0