# Bayesian Active Learning with Fully Bayesian Gaussian Processes

arXiv: https://arxiv.org/abs/2205.10186 · HTML: https://mlsanity.com/paper/2205.10186/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 56 references that are in the map.
- First posted on arXiv 20 May 2022.

## Builds on (most cited first, 13 of 13)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Active Learning for Deep Gaussian Process Surrogates](https://mlsanity.com/paper/2012.08015/) (2020)
- [A Benchmark and Comparison of Active Learning for Logistic Regression](https://mlsanity.com/paper/1611.08618/) (2016)
- [Pool-Based Sequential Active Learning for Regression](https://mlsanity.com/paper/1805.04735/) (2018)
- [Approximate Inference for Fully Bayesian Gaussian Process Regression](https://mlsanity.com/paper/1912.13440/) (2019)
- [Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors](https://mlsanity.com/paper/2006.12335/) (2020)
- [Active Learning for Regression Using Greedy Sampling](https://mlsanity.com/paper/1808.04245/) (2018)
- [Active emulation of computer codes with Gaussian processes -- Application to remote sensing](https://mlsanity.com/paper/1912.06552/) (2019)
- [Entropy-based adaptive design for contour finding and estimating reliability](https://mlsanity.com/paper/2105.11357/) (2021)
- [Adaptive quadrature schemes for Bayesian inference via active learning](https://mlsanity.com/paper/2006.00535/) (2020)

## Built on by (most cited first, 26 of 26)

- [Active learning for data streams: a survey](https://mlsanity.com/paper/2302.08893/) (2023)
- [Self-Correcting Bayesian Optimization through Bayesian Active Learning](https://mlsanity.com/paper/2304.11005/) (2023)
- [Robust Gaussian Processes via Relevance Pursuit](https://mlsanity.com/paper/2410.24222/) (2024)
- [SOBER: Highly Parallel Bayesian Optimization and Bayesian Quadrature over Discrete and Mixed Spaces](https://mlsanity.com/paper/2301.11832/) (2023)
- [Adaptive Batch Sizes for Active Learning A Probabilistic Numerics Approach](https://mlsanity.com/paper/2306.05843/) (2023)
- [A Framework for Strategic Discovery of Credible Neural Network Surrogate Models under Uncertainty](https://mlsanity.com/paper/2403.08901/) (2024)
- [Scalable Gaussian Processes with Latent Kronecker Structure](https://mlsanity.com/paper/2506.06895/) (2025)
- [Hierarchical-Hyperplane Kernels for Actively Learning Gaussian Process Models of Nonstationary Systems](https://mlsanity.com/paper/2303.10022/) (2023)
- [X-Factor: Quality Is a Dataset-Intrinsic Property](https://mlsanity.com/paper/2505.22813/) (2025)
- [Learning Power Flow with Confidence: A Probabilistic Guarantee Framework for Voltage Risk](https://mlsanity.com/paper/2308.07867/) (2023)
- [Bayesian Active Learning for Censored Regression](https://mlsanity.com/paper/2402.11973/) (2024)
- [Practical Bayesian Algorithm Execution via Posterior Sampling](https://mlsanity.com/paper/2410.20596/) (2024)
- [Zero-Shot Performance Prediction for Probabilistic Scaling Laws](https://mlsanity.com/paper/2510.16743/) (2025)
- [Informed Initialization for Bayesian Optimization and Active Learning](https://mlsanity.com/paper/2510.23681/) (2025)
- [Targeted active learning for probabilistic models](https://mlsanity.com/paper/2210.12122/) (2022)
- [Global Safe Sequential Learning via Efficient Knowledge Transfer](https://mlsanity.com/paper/2402.14402/) (2024)
- [Future Aware Safe Active Learning of Time Varying Systems using Gaussian Processes](https://mlsanity.com/paper/2405.10581/) (2024)
- [The Speed-up Factor: A Quantitative Multi-Iteration Active Learning Performance Metric](https://mlsanity.com/paper/2602.13359/) (2026)
- [Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity](https://mlsanity.com/paper/2608.24721/) (2026)
- [Bayesian Active Learning By Distribution Disagreement](https://mlsanity.com/paper/2501.01248/) (2025)
- [BLADE: Bayesian Langevin Active Discovery with Replica Exchange for Identification of Complex Systems](https://mlsanity.com/paper/2503.02983/) (2025)
- [Designing an Optimal Sensor Network via Minimizing Information Loss](https://mlsanity.com/paper/2512.05940/) (2025)
- [CAAL: Confidence-Aware Active Learning for Heteroscedastic Atmospheric Regression](https://mlsanity.com/paper/2602.11825/) (2026)
- [Quality-Controlled Active Learning via Gaussian Processes for Robust Structure-Property Learning in Autonomous Microscopy](https://mlsanity.com/paper/2603.29135/) (2026)
- [The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification](https://mlsanity.com/paper/2606.24903/) (2026)
- [Online Adaptive Kernel Mixing for Gaussian Process Decision Making](https://mlsanity.com/paper/2609.19891/) (2026)

## Abstract (arXiv)

The bias-variance trade-off is a well-known problem in machine learning that only gets more pronounced the less available data there is. In active learning, where labeled data is scarce or difficult to obtain, neglecting this trade-off can cause inefficient and non-optimal querying, leading to unnecessary data labeling. In this paper, we focus on active learning with Gaussian Processes (GPs). For the GP, the bias-variance trade-off is made by optimization of the two hyperparameters: the length scale and noise-term. Considering that the optimal mode of the joint posterior of the hyperparameters is equivalent to the optimal bias-variance trade-off, we approximate this joint posterior and utilize it to design two new acquisition functions. The first one is a Bayesian variant of Query-by-Committee (B-QBC), and the second is an extension that explicitly minimizes the predictive variance through a Query by Mixture of Gaussian Processes (QB-MGP) formulation. Across six simulators, we empirically show that B-QBC, on average, achieves the best marginal likelihood, whereas QB-MGP achieves the best predictive performance. We show that incorporating the bias-variance trade-off in the acquisition functions mitigates unnecessary and expensive data labeling.