mlsanityData to 7 Oct 2026

Random Feature Representation Boosting

arXiv 2501.18283 ↗First posted 30 Jan 2025stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce Random Feature Representation Boosting (RFRBoost), a novel method for constructing deep residual random feature neural networks (RFNNs) using boosting theory. RFRBoost uses random features at each layer to learn the functional gradient of the network representation, enhancing performance while preserving the convex optimization benefits of RFNNs. In the case of MSE loss, we obtain closed-form solutions to greedy layer-wise boosting with random features. For general loss functions, we show that fitting random feature residual blocks reduces to solving a quadratically constrained least squares problem. Through extensive numerical experiments on tabular datasets for both regression and classification, we show that RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in the small- to medium-scale regime where RFNNs are typically applied. Moreover, RFRBoost offers substantial computational benefits, and theoretical guarantees stemming from boosting theory.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 47 of its 89 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

47 of 89 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2019Augmented Neural ODEscited by 396
  9. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  10. 2019ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernelscited by 336
  11. 2017OpenML Benchmarking Suitescited by 169
  12. 2016Generalization Properties of Learning with Random Featurescited by 163
35 more
  1. 2021Neural SDEs as Infinite-Dimensional GANscited by 159
  2. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  3. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  4. 2020The Random Feature Model for Input-Output Maps between Banach Spacescited by 90
  5. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  6. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  7. 2022HYDRA: Competing convolutional kernels for fast and accurate time series classificationcited by 53
  8. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  9. 2024aeon: a Python toolkit for learning from time seriescited by 46
  10. 2017Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systemscited by 39
  11. 2015Optimal Rates for Random Fourier Featurescited by 38
  12. 2021Momentum Residual Neural Networkscited by 36
  13. 2021Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionalitycited by 34
  14. 2024Log Neural Controlled Differential Equations: The Lie Brackets Make a Differencecited by 34
  15. 2018Learning with SGD and Random Featurescited by 33
  16. 2018Scattering Networks for Hybrid Representation Learningcited by 33
  17. 2017Learning Deep ResNet Blocks Sequentially using Boosting Theorycited by 32
  18. 2018But How Does It Work in Theory? Linear SVM with Random Featurescited by 28
  19. 2021Optimal Stopping via Randomized Neural Networkscited by 26
  20. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  21. 2023Error Bounds for Learning with Vector-Valued Random Featurescited by 23
  22. 2023Sampling weights of deep neural networkscited by 20
  23. 2022Understanding the Covariance Structure of Convolutional Filterscited by 16
  24. 2018Functional Gradient Boosting based on Residual Network Perceptioncited by 14
  25. 2023Universal approximation property of Banach space-valued random feature models including random neural networkscited by 12
  26. 2024Random Representations Outperform Online Continually Learned Representationscited by 10
  27. 2022Boosted Dynamic Neural Networkscited by 8
  28. 2023A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regressioncited by 8
  29. 2017Visualizing and Improving Scattering Networkscited by 7
  30. 2023Infinite-dimensional reservoir computingcited by 7
  31. 2024Random features models: a way to study the success of naive imputationcited by 7
  32. 2018On Kernel Derivative Approximation with Random Fourier Featurescited by 6
  33. 2019Sequential Training of Neural Networks with Gradient Boostingcited by 6
  34. 2024Optimal Kernel Quantile Learning with Random Featurescited by 4
  35. 2024RandNet-Parareal: a time-parallel PDE solver using Random Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Rapid training of Hamiltonian graph networks using random featurescited by 2
  2. 2026DeepAFL: Deep Analytic Federated Learningcited by 1
  3. 2026Gradient Regularized Newton Boosting Trees with Global Convergencecited by 0
  4. 2026Random Neural Network Expressivity for Non-Linear Partial Differential Equationscited by 0