mlsanityData to 7 Oct 2026

Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

arXiv 2410.09239 ↗First posted 11 Oct 2024cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

A key task in AutoML is to model learning curves of machine learning models jointly as a function of model hyper-parameters and training progression. While Gaussian processes (GPs) are suitable for this task, na"ive GPs require O(n3m3) time and O(n2 m2) space for n hyper-parameter configurations and O(m) learning curve observations per hyper-parameter. Efficient inference via Kronecker structure is typically incompatible with early-stopping due to missing learning curve values. We impose textitlatent Kronecker structure to leverage efficient product kernels while handling missing values. In particular, we interpret the joint covariance matrix of observed values as the projection of a latent Kronecker product. Combined with iterative linear solvers and structured matrix-vector multiplication, our method only requires O(n3 + m3) time and O(n2 + m2) space. We show that our GP model can match the performance of a Transformer on a learning curve prediction task.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

15 of 22 references.

References · 15

  1. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  2. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 175
  3. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 125
  4. 2019Exact Gaussian Processes on a Million Data Pointscited by 118
  5. 2024Vanilla Bayesian Optimization Performs Great in High Dimensionscited by 87
  6. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  7. 2020Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDLcited by 67
  8. 2020Pathwise Conditioning of Gaussian Processescited by 63
  9. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  10. 2021Bayesian Optimization with High-Dimensional Outputscited by 32
  11. 2023Sampling from Gaussian Process Posteriors using Stochastic Gradient Descentcited by 27
  12. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
3 more
  1. 2023Stochastic Gradient Descent for Gaussian Processes Done Rightcited by 16
  2. 2022Supervising the Multi-Fidelity Race of Hyperparameter Configurationscited by 15
  3. 2024Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 22
  2. 2024Robust Gaussian Processes via Relevance Pursuitcited by 7
  3. 2025Graph Random Features for Scalable Gaussian Processescited by 4
  4. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  5. 2025Derivation of Output Correlation Inferences for Multi-Output (aka Multi-Task) Gaussian Processcited by 1
  6. 2025Large Language Model as Meta-Surrogate for Data-Driven Many-Task Optimization: A Proof-of-Principle Studycited by 1
  7. 2025Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processescited by 1
  8. 2026Spatio-Temporal Gaussian Process for Building Terrain-Incorporating Wind Power Curvescited by 1
  9. 2026Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruningcited by 0