mlsanityData to 14 Oct 2026

Generative Feature Training of Thin 2-Layer Networks

arXiv 2411.06848 ↗First posted 11 Nov 2024cs.LG cs.NA math.NAcited by 2

Not on the map. Why?

Abstract (arXiv)

We consider the approximation of functions by 2-layer neural networks with a small number of hidden weights based on the squared loss and small datasets. Due to the highly non-convex energy landscape, gradient-based training often suffers from local minima. As a remedy, we initialize the hidden weights with samples from a learned proposal distribution, which we parameterize as a deep generative model. To train this model, we exploit the fact that with fixed hidden weights, the optimal output weights solve a linear equation. After learning the generative model, we refine the sampled weights with a gradient-based post-processing in the latent space. Here, we also include a regularization scheme to counteract potential noise. Finally, we demonstrate the effectiveness of our approach by numerical examples.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

25 of 44 references.

References · 25

  1. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
  2. 2016Generalization Properties of Learning with Random Featurescited by 164
  3. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
  4. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 109
  5. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  6. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  7. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  8. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  9. 2018Complexity of Training ReLU Neural Networkcited by 23
  10. 2021Generalization Bounds for Sparse Random Feature Expansionscited by 22
  11. 2023Sampling weights of deep neural networkscited by 20
  12. 2022Efficient Global Optimization of Two-Layer ReLU Networks: Quadratic-Time Algorithms and Adversarial Trainingcited by 15
13 more
  1. 2019Implicit Kernel Learningcited by 13
  2. 2022HARFE: Hard-Ridge Random Feature Expansioncited by 12
  3. 2021Interpretable Approximation of High-Dimensional Datacited by 10
  4. 2021SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruningcited by 9
  5. 2016Additive Approximations in High Dimensional Nonparametric Regression via the SALSAcited by 8
  6. 2024Fast Kernel Summation in High Dimensions via Slicing and Fourier Transformscited by 8
  7. 2024Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Featurescited by 7
  8. 2024Fast Summation of Radial Kernels via QMC Slicingcited by 7
  9. 2020Training Two-Layer ReLU Networks with Gradient Descent is Inconsistentcited by 6
  10. 2021Generalization Error Analysis of Neural networks with Gradient Based Regularizationcited by 3
  11. 2024ANOVA-boosting for Random Fourier Featurescited by 2
  12. 2023Training a Two Layer ReLU Network Analyticallycited by 1
  13. 2024Sparse additive function decompositions facing basis transformscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Smoothed Distance Kernels for MMDs and Applications in Wasserstein Gradient Flowscited by 2
  2. 2026Learning solution operators of PDEs with sparse approximation methodscited by 0