# Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities

arXiv: https://arxiv.org/abs/2208.00549 · HTML: https://mlsanity.com/paper/2208.00549/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Submodular Combinatorial Information Measures with Applications in Machine Learning](https://mlsanity.com/paper/2006.15412/) (2020)
- [Gone Fishing: Neural Active Learning with Fisher Embeddings](https://mlsanity.com/paper/2106.09675/) (2021)
- [SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios](https://mlsanity.com/paper/2107.00717/) (2021)
- [PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Subset Selection](https://mlsanity.com/paper/2103.00128/) (2021)
- [Information-theoretic Online Memory Selection for Continual Learning](https://mlsanity.com/paper/2204.04763/) (2022)
- [Stochastic Batch Acquisition: A Simple Baseline for Deep Active Learning](https://mlsanity.com/paper/2106.12059/) (2021)
- [Convergence Rates of Active Learning for Maximum Likelihood Estimation](https://mlsanity.com/paper/1506.02348/) (2015)
- [Active Learning for Speech Recognition: the Power of Gradients](https://mlsanity.com/paper/1612.03226/) (2016)
- [Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers](https://mlsanity.com/paper/2106.11719/) (2021)
- [Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?](https://mlsanity.com/paper/2011.03178/) (2020)
- [A Practical & Unified Notation for Information-Theoretic Quantities in ML](https://mlsanity.com/paper/2106.12062/) (2021)

## Built on by (most cited first, 28 of 28)

- [PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting](https://mlsanity.com/paper/2406.10219/) (2024)
- [FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher Information](https://mlsanity.com/paper/2311.17874/) (2023)
- [GaussianMarker: Uncertainty-Aware Copyright Protection of 3D Gaussian Splatting](https://mlsanity.com/paper/2410.23718/) (2024)
- [POp-GS: Next Best View in 3D-Gaussian Splatting with P-Optimality](https://mlsanity.com/paper/2503.07819/) (2025)
- [Black-Box Batch Active Learning for Regression](https://mlsanity.com/paper/2302.08981/) (2023)
- [Next Best Sense: Guiding Vision and Touch with FisherRF for 3D Gaussian Splatting](https://mlsanity.com/paper/2410.04680/) (2024)
- [MarkSplatter: Generalizable Watermarking for 3D Gaussian Splatting Model via Splatter Image Structure](https://mlsanity.com/paper/2509.00757/) (2025)
- [OASIS: Online Sample Selection for Continual Visual Instruction Tuning](https://mlsanity.com/paper/2506.02011/) (2025)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [Active Preference Learning for Ordering Items In- and Out-of-sample](https://mlsanity.com/paper/2405.03059/) (2024)
- [Active Learning for Continual Learning: Keeping the Past Alive in the Present](https://mlsanity.com/paper/2501.14278/) (2025)
- [Uncertainty-driven 3D Gaussian Splatting Active Mapping via Anisotropic Visibility Field](https://mlsanity.com/paper/2605.30342/) (2026)
- [ATLAS: Active Theory Learning for Automated Science](https://mlsanity.com/paper/2606.12386/) (2026)
- [FisherMask: Enhancing Neural Network Labeling Efficiency in Image Classification Using Fisher Information](https://mlsanity.com/paper/2411.05752/) (2024)
- [Distributionally Robust Active Learning for Gaussian Process Regression](https://mlsanity.com/paper/2502.16870/) (2025)
- [BatchGFN: Generative Flow Networks for Batch Active Learning](https://mlsanity.com/paper/2306.15058/) (2023)
- [Bayesian Active Learning for Censored Regression](https://mlsanity.com/paper/2402.11973/) (2024)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)
- [On the Convergence of Loss and Uncertainty-based Active Learning Algorithms](https://mlsanity.com/paper/2312.13927/) (2023)
- [Information-Theoretic Active Correlation Clustering](https://mlsanity.com/paper/2402.03587/) (2024)
- [FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain](https://mlsanity.com/paper/2511.21113/) (2025)
- [Active Learning Using Aggregated Acquisition Functions: Accuracy and Sustainability Analysis](https://mlsanity.com/paper/2602.07440/) (2026)
- [MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization](https://mlsanity.com/paper/2605.10784/) (2026)
- [Optimal and Efficient Binary Questioning for Human-in-the-Loop Annotation](https://mlsanity.com/paper/2307.01578/) (2023)
- [Prediction-Oriented Subsampling from Data Streams](https://mlsanity.com/paper/2508.03868/) (2025)
- [When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoff](https://mlsanity.com/paper/2509.04363/) (2025)
- [From Uncertainty to Stability and Fidelity: Guiding Sparse-View 3D Gaussian Splatting with Fisher Information](https://mlsanity.com/paper/2606.20842/) (2026)
- [AGILE-GS: Anchor-Guided Fast Next-Best-View Selection for Active 3D Gaussian Splatting](https://mlsanity.com/paper/2609.34176/) (2026)

## Abstract (arXiv)

Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based on gradients, and gradient lengths to estimate how informative data is for a model's training. Are these different approaches connected, and if so, how? We revisit the fundamentals of Bayesian optimal experiment design and show that these recently proposed methods can be understood as approximations to information-theoretic quantities: among them, the mutual information between predictions and model parameters, known as expected information gain or BALD in machine learning, and the mutual information between predictions of acquisition candidates and test samples, known as expected predictive information gain. We develop a comprehensive set of approximations using Fisher information and observed information and derive a unified framework that connects seemingly disparate literature. Although Bayesian methods are often seen as separate from non-Bayesian ones, the sometimes fuzzy notion of "informativeness" expressed in various non-Bayesian objectives leads to the same couple of information quantities, which were, in principle, already known by Lindley (1956) and MacKay (1992).