# Direct loss minimization algorithms for sparse Gaussian processes

arXiv: https://arxiv.org/abs/2004.03083 · HTML: https://mlsanity.com/paper/2004.03083/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference](https://mlsanity.com/paper/2411.01036/) (2024)
- [Approximation-Aware Bayesian Optimization](https://mlsanity.com/paper/2406.04308/) (2024)
- [Scalable Cross Validation Losses for Gaussian Process Models](https://mlsanity.com/paper/2105.11535/) (2021)
- [Loss-calibrated expectation propagation for approximate Bayesian decision-making](https://mlsanity.com/paper/2201.03128/) (2022)
- [On the Performance of Direct Loss Minimization for Bayesian Neural Networks](https://mlsanity.com/paper/2211.08393/) (2022)
- [Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Mining](https://mlsanity.com/paper/2308.03035/) (2023)
- [Decision-aware training of spatiotemporal forecasting models to select a top K subset of sites for intervention](https://mlsanity.com/paper/2503.05622/) (2025)
- [Variational Inference on the Final-Layer Output of Neural Networks](https://mlsanity.com/paper/2302.02420/) (2023)
- [Stability-based Generalization Bounds for Variational Inference](https://mlsanity.com/paper/2502.12353/) (2025)
- [Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding](https://mlsanity.com/paper/2605.08446/) (2026)

## Abstract (arXiv)

The paper provides a thorough investigation of Direct loss minimization (DLM), which optimizes the posterior to minimize predictive loss, in sparse Gaussian processes. For the conjugate case, we consider DLM for log-loss and DLM for square loss showing a significant performance improvement in both cases. The application of DLM in non-conjugate cases is more complex because the logarithm of expectation in the log-loss DLM objective is often intractable and simple sampling leads to biased estimates of gradients. The paper makes two technical contributions to address this. First, a new method using product sampling is proposed, which gives unbiased estimates of gradients (uPS) for the objective function. Second, a theoretical analysis of biased Monte Carlo estimates (bMC) shows that stochastic gradient descent converges despite the biased gradients. Experiments demonstrate empirical success of DLM. A comparison of the sampling methods shows that, while uPS is potentially more sample-efficient, bMC provides a better tradeoff in terms of convergence time and computational efficiency.