# Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

arXiv: https://arxiv.org/abs/2410.09239 · HTML: https://mlsanity.com/paper/2410.09239/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 15 of 15)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [Efficiently Sampling Functions from Gaussian Process Posteriors](https://mlsanity.com/paper/2002.09309/) (2020)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Vanilla Bayesian Optimization Performs Great in High Dimensions](https://mlsanity.com/paper/2402.02229/) (2024)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL](https://mlsanity.com/paper/2006.13799/) (2020)
- [Pathwise Conditioning of Gaussian Processes](https://mlsanity.com/paper/2011.04026/) (2020)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [Bayesian Optimization with High-Dimensional Outputs](https://mlsanity.com/paper/2106.12997/) (2021)
- [Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent](https://mlsanity.com/paper/2306.11589/) (2023)
- [In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2404.16795/) (2024)
- [Stochastic Gradient Descent for Gaussian Processes Done Right](https://mlsanity.com/paper/2310.20581/) (2023)
- [Supervising the Multi-Fidelity Race of Hyperparameter Configurations](https://mlsanity.com/paper/2202.09774/) (2022)
- [Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes](https://mlsanity.com/paper/2405.18457/) (2024)

## Built on by (most cited first, 10 of 10)

- [Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration](https://mlsanity.com/paper/2512.22382/) (2025)
- [Robust Gaussian Processes via Relevance Pursuit](https://mlsanity.com/paper/2410.24222/) (2024)
- [Graph Random Features for Scalable Gaussian Processes](https://mlsanity.com/paper/2509.03691/) (2025)
- [Zero-Shot Performance Prediction for Probabilistic Scaling Laws](https://mlsanity.com/paper/2510.16743/) (2025)
- [Derivation of Output Correlation Inferences for Multi-Output (aka Multi-Task) Gaussian Process](https://mlsanity.com/paper/2501.07964/) (2025)
- [Large Language Model as Meta-Surrogate for Data-Driven Many-Task Optimization: A Proof-of-Principle Study](https://mlsanity.com/paper/2503.08301/) (2025)
- [Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes](https://mlsanity.com/paper/2508.14818/) (2025)
- [Spatio-Temporal Gaussian Process for Building Terrain-Incorporating Wind Power Curves](https://mlsanity.com/paper/2607.00051/) (2026)
- [Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning](https://mlsanity.com/paper/2605.17234/) (2026)
- [Target-Constrained Pooling for Trajectory Extrapolation](https://mlsanity.com/paper/2610.13034/) (2026)

## Abstract (arXiv)

A key task in AutoML is to model learning curves of machine learning models jointly as a function of model hyper-parameters and training progression. While Gaussian processes (GPs) are suitable for this task, na"ive GPs require O(n3m3) time and O(n2 m2) space for n hyper-parameter configurations and O(m) learning curve observations per hyper-parameter. Efficient inference via Kronecker structure is typically incompatible with early-stopping due to missing learning curve values. We impose textitlatent Kronecker structure to leverage efficient product kernels while handling missing values. In particular, we interpret the joint covariance matrix of observed values as the projection of a latent Kronecker product. Combined with iterative linear solvers and structured matrix-vector multiplication, our method only requires O(n3 + m3) time and O(n2 + m2) space. We show that our GP model can match the performance of a Transformer on a learning curve prediction task.