A Practical & Unified Notation for Information-Theoretic Quantities in ML
Not on the map. Why?
Abstract (arXiv)
A practical notation can convey valuable intuitions and concisely express new ideas. Information theory is of importance to machine learning, but the notation for information-theoretic quantities is sometimes opaque. We propose a practical and unified notation and extend it to include information-theoretic quantities between observed outcomes (events) and random variables. This includes the point-wise mutual information known in NLP and mixed quantities such as specific surprise and specific information in the cognitive sciences and information gain in Bayesian optimal experimental design. We apply our notation to prove a version of Stirling's approximation for binomial coefficients mentioned by MacKa (2003) using new intuitions. We also concisely rederive the evidence lower bound for variational auto-encoders and variational inference in approximate Bayesian neural networks. Furthermore, we apply the notation to a popular information-theoretic acquisition function in Bayesian active learning which selects the most informative (unlabelled) samples to be labelled by an expert and extend this acquisition function to the core-set problem with the goal of selecting the most informative samples given the labels.
Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 9 of its 29 references that are in the map. First posted on arXiv 22 Jun 2021.
Builds on
9 of 29 references.
References · 9
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,430
- 2017Deep Bayesian Active Learning with Image Datacited by 816
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 497
- 2015Variational Dropout and the Local Reparameterization Trickcited by 490
- 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 430
- 2020A Theory of Usable Information Under Computational Constraintscited by 156
- 2019Variational Bayesian Optimal Experimental Designcited by 116
- 2020Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learningcited by 16
Built on by
Citing papers per year: 2022: 3; 2023: 1.
4 papers.
- 2022Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantitiescited by 28
- 2023Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessmentcited by 26
- 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
- 2022Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Samplingcited by 3
- 2023Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment25 Jan 2023
- 2022Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities1 Aug 2022
- 2022Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Sampling18 May 2022
- 2022A Note on "Assessing Generalization of SGD via Disagreement"3 Feb 2022