# Prediction-Oriented Bayesian Active Learning

arXiv: https://arxiv.org/abs/2304.08151 · HTML: https://mlsanity.com/paper/2304.08151/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 81 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [Deep Active Learning for Named Entity Recognition](https://mlsanity.com/paper/1707.05928/) (2017)
- [Noisy Natural Gradient as Variational Inference](https://mlsanity.com/paper/1712.02390/) (2017)
- [Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study](https://mlsanity.com/paper/1808.05697/) (2018)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Bayesian Batch Active Learning as Sparse Subset Approximation](https://mlsanity.com/paper/1908.02144/) (2019)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [Bayesian Generative Active Deep Learning](https://mlsanity.com/paper/1904.11643/) (2019)
- [Towards Robust and Reproducible Active Learning Using Neural Networks](https://mlsanity.com/paper/2002.09564/) (2020)
- [Differentiable Compositional Kernel Learning for Gaussian Processes](https://mlsanity.com/paper/1806.04326/) (2018)
- [Diversity Enhanced Active Learning with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2110.14171/) (2021)
- [Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?](https://mlsanity.com/paper/2011.03178/) (2020)
- [Active Learning for Decision-Making from Imbalanced Observational Data](https://mlsanity.com/paper/1904.05268/) (2019)
- [Bayesian active learning for production, a systematic study and a reusable library](https://mlsanity.com/paper/2006.09916/) (2020)
- [Targeted Active Learning for Bayesian Decision-Making](https://mlsanity.com/paper/2106.04193/) (2021)

## Built on by (most cited first, 30 of 62)

- [Rethinking Aleatoric and Epistemic Uncertainty](https://mlsanity.com/paper/2412.20892/) (2024)
- [BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design](https://mlsanity.com/paper/2508.21184/) (2025)
- [Amortized Bayesian Experimental Design for Decision-Making](https://mlsanity.com/paper/2411.02064/) (2024)
- [CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training](https://mlsanity.com/paper/2406.10670/) (2024)
- [Deep Bayesian Active Learning for Preference Modeling in Large Language Models](https://mlsanity.com/paper/2406.10023/) (2024)
- [On Subjective Uncertainty Quantification and Calibration in Natural Language Generation](https://mlsanity.com/paper/2406.05213/) (2024)
- [Post-hoc Probabilistic Vision-Language Models](https://mlsanity.com/paper/2412.06014/) (2024)
- [DiscoBAX: Discovery of Optimal Intervention Sets in Genomic Experiment Design](https://mlsanity.com/paper/2312.04064/) (2023)
- [Nesting Particle Filters for Experimental Design in Dynamical Systems](https://mlsanity.com/paper/2402.07868/) (2024)
- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)
- [Active Fine-Tuning of Multi-Task Policies](https://mlsanity.com/paper/2410.05026/) (2024)
- [Variational Sequential Optimal Experimental Design using Reinforcement Learning](https://mlsanity.com/paper/2306.10430/) (2023)
- [ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition](https://mlsanity.com/paper/2506.07259/) (2025)
- [Deep Stochastic Processes via Functional Markov Transition Operators](https://mlsanity.com/paper/2305.15574/) (2023)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models](https://mlsanity.com/paper/2608.09696/) (2026)
- [Streamlining Prediction in Bayesian Deep Learning](https://mlsanity.com/paper/2411.18425/) (2024)
- [Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty](https://mlsanity.com/paper/2412.08225/) (2024)
- [Active Learning with Selective Time-Step Acquisition for PDEs](https://mlsanity.com/paper/2511.18107/) (2025)
- [Distributionally Robust Active Learning for Gaussian Process Regression](https://mlsanity.com/paper/2502.16870/) (2025)
- [Distributional Training Data Attribution: What do Influence Functions Sample?](https://mlsanity.com/paper/2506.12965/) (2025)
- [Cold-Start Personalization via Training-Free Priors from Structured World Models](https://mlsanity.com/paper/2602.15012/) (2026)
- [Enhancing Trustworthiness in ML-Based Network Intrusion Detection with Uncertainty Quantification](https://mlsanity.com/paper/2310.10655/) (2023)
- [Bayesian Active Learning for Censored Regression](https://mlsanity.com/paper/2402.11973/) (2024)
- [Active Few-Shot Fine-Tuning](https://mlsanity.com/paper/2402.15441/) (2024)
- [Goal-Oriented Sequential Bayesian Experimental Design for Causal Learning](https://mlsanity.com/paper/2507.07359/) (2025)
- [BioBO: Biology-informed Bayesian Optimization for Perturbation Design](https://mlsanity.com/paper/2509.19988/) (2025)
- [Causal-EPIG: A Prediction-Oriented Active Learning Framework for CATE Estimation](https://mlsanity.com/paper/2509.21866/) (2025)
- [Accelerated Learning on Large Scale Screens using Generative Library Models](https://mlsanity.com/paper/2510.16612/) (2025)
- [Deep Ensembles for Epistemic Uncertainty: A Frequentist Perspective](https://mlsanity.com/paper/2510.22063/) (2025)

## Abstract (arXiv)

Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BALD score. We highlight that this can be suboptimal from the perspective of predictive performance. For example, BALD lacks a notion of an input distribution and so is prone to prioritise data of limited relevance. To address this we propose the expected predictive information gain (EPIG), an acquisition function that measures information gain in the space of predictions rather than parameters. We find that using EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models, and thus provides an appealing drop-in replacement.