On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions
Not on the map. Why?
Abstract (arXiv)
This paper studies the generalization properties of a recently proposed kernel method, the Random Feature models with Learnable Activation Functions (RFLAF). By applying a data-dependent sampling scheme for generating features, we provide by far the sharpest bounds on the required number of features for learning RFLAF in both the regression and classification tasks. We provide a unified theorem that describes the complexity of the feature number s, and discuss the results for the plain sampling scheme and the data-dependent leverage weighted scheme. Through weighted sampling, the bound on s in the MSE loss case is improved from Omega(1/ε2) to tildeOmega((1/ε)1/t) in general (t≥ 1), and even to Omega(1) when the Gram matrix has a finite rank. For the Lipschitz loss case, the bound is improved from Omega(1/ε2) to tildeOmega((1/ε2)1/t). To learn the weighted RFLAF, we also propose an algorithm to find an approximate kernel and then apply the leverage weighted sampling. Empirical results show that the weighted RFLAF achieves the same performances with a significantly fewer number of features compared to the plainly sampled RFLAF, validating our theories and the effectiveness of this method.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 17 Oct 2025.
Builds on
32 of 55 references.
References · 32
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2024KAN: Kolmogorov-Arnold Networkscited by 634
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2015An Introduction to Matrix Concentration Inequalitiescited by 391
- 2021Random Feature Attentioncited by 350
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
- 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
- 2016Generalization Properties of Learning with Random Featurescited by 163
- 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
- 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
20 more
- 2024KAN 2.0: Kolmogorov-Arnold Networks Meet Sciencecited by 116
- 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
- 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
- 2015Less is More: Nystr"om Computational Regularizationcited by 89
- 2017FALKON: An Optimal Large Scale Kernel Methodcited by 82
- 2015On the Error of Random Fourier Featurescited by 76
- 2018Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guaranteescited by 71
- 2013Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Ratescited by 64
- 2020A law of robustness for two-layers neural networkscited by 54
- 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
- 2015Optimal Rates for Random Fourier Featurescited by 38
- 2019Learning Activation Functions: A new paradigm for understanding Neural Networkscited by 33
- 2022ExSpliNet: An interpretable and expressive spline-based neural networkcited by 33
- 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
- 2017Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamicscited by 28
- 2018But How Does It Work in Theory? Linear SVM with Random Featurescited by 28
- 2020SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustnesscited by 18
- 2022Neural Network Architecture Beyond Width and Depthcited by 17
- 2021Learning specialized activation functions with the Piecewise Linear Unitcited by 7
- 2025SchoenbAt: Rethinking Attention with Polynomial basiscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.