mlsanityData to 7 Oct 2026

Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features

arXiv 2407.00584 ↗First posted 30 Jun 2024cs.LG stat.CO stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Randomized algorithms exploit stochasticity to reduce computational complexity. One important example is random feature regression (RFR) that accelerates Gaussian process regression (GPR). RFR approximates an unknown function with a random neural network whose hidden weights and biases are sampled from a probability distribution. Only the final output layer is fit to data. In randomized algorithms like RFR, the hyperparameters that characterize the sampling distribution greatly impact performance, yet are not directly accessible from samples. This makes optimization of hyperparameters via standard (gradient-based) optimization tools inapplicable. Inspired by Bayesian ideas from GPR, this paper introduces a random objective function that is tailored for hyperparameter tuning of vector-valued random features. The objective is minimized with ensemble Kalman inversion (EKI). EKI is a gradient-free particle-based optimizer that is scalable to high-dimensions and robust to randomness in objective functions. A numerical study showcases the new black-box methodology to learn hyperparameter distributions in several problems that are sensitive to the hyperparameter selection: two global sensitivity analyses, integrating a chaotic dynamical system, and solving a Bayesian inverse problem from atmospheric dynamics. The success of the proposed EKI-based algorithm for RFR suggests its potential for automated optimization of hyperparameters arising in other randomized algorithms.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 36 of its 100 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

36 of 100 references.

References · 36

  1. 2017Large-Scale Evolution of Image Classifierscited by 348
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  3. 2014Scalable Variational Gaussian Process Classificationcited by 226
  4. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  5. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  6. 2016Generalization Properties of Learning with Random Featurescited by 163
  7. 2021Next Generation Reservoir Computingcited by 160
  8. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
  9. 2020The Random Feature Model for Input-Output Maps between Banach Spacescited by 90
  10. 2024Vanilla Bayesian Optimization Performs Great in High Dimensionscited by 87
  11. 2016Variational Fourier features for Gaussian processescited by 85
  12. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 72
24 more
  1. 2015Operator-valued Kernels for Learning from Functional Response Datacited by 59
  2. 2018Kernel Flows: from learning kernels from data into the abysscited by 49
  3. 2019Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computerscited by 43
  4. 2018Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Taskscited by 36
  5. 2024Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimizationcited by 33
  6. 2021Bayesian Optimization with High-Dimensional Outputscited by 32
  7. 2021Robust Optimization and Validation of Echo State Networks for learning chaotic dynamicscited by 27
  8. 2023Error Bounds for Learning with Vector-Valued Random Featurescited by 23
  9. 2016Random Fourier Features for Operator-Valued Kernelscited by 22
  10. 2022Ensemble Kalman Methods: A Mean Field Perspectivecited by 20
  11. 2014A la Carte - Learning Fast Kernelscited by 18
  12. 2017The prior can generally only be understood in the context of the likelihoodcited by 18
  13. 2018Inverse Problems and Data Assimilationcited by 18
  14. 2020Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimationcited by 18
  15. 2020Do ideas have shape? Idea registration as the continuous limit of artificial neural networkscited by 16
  16. 2019Implicit Kernel Learningcited by 13
  17. 2020Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithmcited by 13
  18. 2023Learning About Structural Errors in Models of Complex Dynamical Systemscited by 13
  19. 2013Compact Random Feature Mapscited by 12
  20. 2022Efficient Hyperparameter Tuning for Large Scale Kernel Ridge Regressioncited by 11
  21. 2023Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversioncited by 9
  22. 2021Ensemble Inference Methods for Models With Noisy and Expensive Likelihoodscited by 8
  23. 2020Bayesian optimisation of large-scale photonic reservoir computerscited by 7
  24. 2020Adaptive regularisation for ensemble Kalman inversioncited by 7

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 4.

7 papers.

  1. 2024Operator Learning Using Random Features: A Tool for Scientific Computingcited by 21
  2. 2025Learning dynamical systems with hit-and-run random feature mapscited by 7
  3. 2025Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processescited by 3
  4. 2024On the choice of the non-trainable internal weights in random feature mapscited by 2
  5. 2024Generative Feature Training of Thin 2-Layer Networkscited by 2
  6. 2025Learning where to learn: Training data distribution optimization for scientific machine learningcited by 1
  7. 2025Closed-Form Beta Distribution Estimation from Sparse Statistics with Random Forest Implicit Regularizationcited by 0