# An Information-Theoretic Framework for Unifying Active Learning Problems

arXiv: https://arxiv.org/abs/2012.10695 · HTML: https://mlsanity.com/paper/2012.10695/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2020.

## Built on by (most cited first, 14 of 14)

- [Bayesian Optimization under Stochastic Delayed Feedback](https://mlsanity.com/paper/2206.09341/) (2022)
- [Trusted-Maximizers Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/2107.14465/) (2021)
- [All models are wrong, some are useful: Model Selection with Limited Labels](https://mlsanity.com/paper/2410.13609/) (2024)
- [Look-Ahead Acquisition Functions for Bernoulli Level Set Estimation](https://mlsanity.com/paper/2203.09751/) (2022)
- [Training-Free Neural Active Learning with Initialization-Robustness Guarantees](https://mlsanity.com/paper/2306.04454/) (2023)
- [Convolutional Normalizing Flows for Deep Gaussian Processes](https://mlsanity.com/paper/2104.08472/) (2021)
- [Constrained Bayesian Optimization with Adaptive Active Learning of Unknown Constraints](https://mlsanity.com/paper/2310.08751/) (2023)
- [PIED: Physics-Informed Experimental Design for Inverse Problems](https://mlsanity.com/paper/2503.07070/) (2025)
- [Robust Transfer Learning for Active Level Set Estimation with Locally Adaptive Gaussian Process Prior](https://mlsanity.com/paper/2410.05660/) (2024)
- [High-dimensional Level Set Estimation with Trust Regions and Double Acquisition Functions](https://mlsanity.com/paper/2602.12391/) (2026)
- [Nearest Neighbor Classifier with Margin Penalty for Active Learning](https://mlsanity.com/paper/2203.09174/) (2022)
- [An (ε,δ)-accurate level set estimation with a stopping criterion](https://mlsanity.com/paper/2503.20272/) (2025)
- [Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction](https://mlsanity.com/paper/2510.15422/) (2025)
- [STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verification](https://mlsanity.com/paper/2609.23716/) (2026)

## Abstract (arXiv)

This paper presents an information-theoretic framework for unifying active learning problems: level set estimation (LSE), Bayesian optimization (BO), and their generalized variant. We first introduce a novel active learning criterion that subsumes an existing LSE algorithm and achieves state-of-the-art performance in LSE problems with a continuous input domain. Then, by exploiting the relationship between LSE and BO, we design a competitive information-theoretic acquisition function for BO that has interesting connections to upper confidence bound and max-value entropy search (MES). The latter connection reveals a drawback of MES which has important implications on not only MES but also on other MES-based acquisition functions. Finally, our unifying information-theoretic framework can be applied to solve a generalized problem of LSE and BO involving multiple level sets in a data-efficient manner. We empirically evaluate the performance of our proposed algorithms using synthetic benchmark functions, a real-world dataset, and in hyperparameter tuning of machine learning models.