mlsanityData to 7 Oct 2026

Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities

arXiv 2208.00549 ↗First posted 1 Aug 2022cs.LG cs.AI cs.IRcited by 28

Not on the map. Why?

Abstract (arXiv)

Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based on gradients, and gradient lengths to estimate how informative data is for a model's training. Are these different approaches connected, and if so, how? We revisit the fundamentals of Bayesian optimal experiment design and show that these recently proposed methods can be understood as approximations to information-theoretic quantities: among them, the mutual information between predictions and model parameters, known as expected information gain or BALD in machine learning, and the mutual information between predictions of acquisition candidates and test samples, known as expected predictive information gain. We develop a comprehensive set of approximations using Fisher information and observed information and derive a unified framework that connects seemingly disparate literature. Although Bayesian methods are often seen as separate from non-Bayesian ones, the sometimes fuzzy notion of "informativeness" expressed in various non-Bayesian objectives leads to the same couple of information quantities, which were, in principle, already known by Lindley (1956) and MacKay (1992).

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 21 of its 41 references that are in the map. First posted on arXiv 1 Aug 2022.

Builds on

21 of 41 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2017Deep Bayesian Active Learning with Image Datacited by 812
  4. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  5. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  6. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  7. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  8. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  9. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  10. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  11. 2020Submodular Combinatorial Information Measures with Applications in Machine Learningcited by 88
  12. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
9 more
  1. 2021SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarioscited by 83
  2. 2021PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Subset Selectioncited by 50
  3. 2022Information-theoretic Online Memory Selection for Continual Learningcited by 42
  4. 2021Stochastic Batch Acquisition: A Simple Baseline for Deep Active Learningcited by 36
  5. 2015Convergence Rates of Active Learning for Maximum Likelihood Estimationcited by 25
  6. 2016Active Learning for Speech Recognition: the Power of Gradientscited by 25
  7. 2021Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outlierscited by 22
  8. 2020Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?cited by 21
  9. 2021A Practical & Unified Notation for Information-Theoretic Quantities in MLcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 9; 2026: 6.

28 papers.

Sort:

By area:

  1. 2024PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splattingcited by 85
  2. 2023FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher Informationcited by 63
  3. 2024GaussianMarker: Uncertainty-Aware Copyright Protection of 3D Gaussian Splattingcited by 20
  4. 2025POp-GS: Next Best View in 3D-Gaussian Splatting with P-Optimalitycited by 18
  5. 2023Black-Box Batch Active Learning for Regressioncited by 12
  6. 2024Next Best Sense: Guiding Vision and Touch with FisherRF for 3D Gaussian Splattingcited by 10
  7. 2025MarkSplatter: Generalizable Watermarking for 3D Gaussian Splatting Model via Splatter Image Structurecited by 10
  8. 2025OASIS: Online Sample Selection for Continual Visual Instruction Tuningcited by 8
  9. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  10. 2024Active Preference Learning for Ordering Items In- and Out-of-samplecited by 6
  11. 2025Active Learning for Continual Learning: Keeping the Past Alive in the Presentcited by 5
  12. 2026Uncertainty-driven 3D Gaussian Splatting Active Mapping via Anisotropic Visibility Fieldcited by 4
16 more
  1. 2024FisherMask: Enhancing Neural Network Labeling Efficiency in Image Classification Using Fisher Informationcited by 3
  2. 2025Distributionally Robust Active Learning for Gaussian Process Regressioncited by 3
  3. 2026ATLAS: Active Theory Learning for Automated Sciencecited by 3
  4. 2023BatchGFN: Generative Flow Networks for Batch Active Learningcited by 2
  5. 2024Bayesian Active Learning for Censored Regressioncited by 2
  6. 2025Effective Data Pruning through Score Extrapolationcited by 2
  7. 2023On the Convergence of Loss and Uncertainty-based Active Learning Algorithmscited by 1
  8. 2024Information-Theoretic Active Correlation Clusteringcited by 1
  9. 2025FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gaincited by 1
  10. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  11. 2023Optimal and Efficient Binary Questioning for Human-in-the-Loop Annotationcited by 0
  12. 2025Prediction-Oriented Subsampling from Data Streamscited by 0
  13. 2025When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoffcited by 0
  14. 2026Active Learning Using Aggregated Acquisition Functions: Accuracy and Sustainability Analysiscited by 0
  15. 2026From Uncertainty to Stability and Fidelity: Guiding Sparse-View 3D Gaussian Splatting with Fisher Informationcited by 0
  16. 2026AGILE-GS: Anchor-Guided Fast Next-Best-View Selection for Active 3D Gaussian Splattingcited by 0