Laplace Sample Information: Data Informativeness Through a Bayesian Lens
Not on the map. Why?
Abstract (arXiv)
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 56 of its 75 references that are in the map. First posted on arXiv 21 May 2025.
Builds on
56 of 75 references.
References · 56
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2016Deep Learning with Differential Privacycited by 2,728
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2017Renyi Differential Privacycited by 650
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
38 more
- 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
- 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
- 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
- 2023Studying Large Language Model Generalization with Influence Functionscited by 326
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
- 2023TRAK: Attributing Model Behavior at Scalecited by 283
- 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
- 2020Influence Functions in Deep Learning Are Fragilecited by 252
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
- 2023Data-centric Artificial Intelligence: A Surveycited by 203
- 2019What Do Compressed Deep Neural Networks Forget?cited by 171
- 2020Characterising Bias in Compressed Modelscited by 167
- 2020A Theory of Usable Information Under Computational Constraintscited by 162
- 2020DeltaGrad: Rapid retraining of machine learning modelscited by 149
- 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
- 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
- 2021Scaling Up Influence Functionscited by 132
- 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
- 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
- 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
- 2015Algorithmic Stability for Adaptive Data Analysiscited by 88
- 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 83
- 2023Data Pruning via Moving-one-Sample-outcited by 67
- 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
- 2021What can linearized neural networks actually say about generalization?cited by 49
- 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
- 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
- 2022Private Set Generation with Discriminative Informationcited by 34
- 2019The importance of better models in stochastic optimizationcited by 32
- 2017Calibrating Noise to Variance in Adaptive Data Analysiscited by 29
- 2023Theoretical and Practical Perspectives on what Influence Functions Docited by 29
- 2016Max-Information, Differential Privacy, and Post-Selection Hypothesis Testingcited by 27
- 2022Individual Privacy Accounting for Differentially Private Stochastic Gradient Descentcited by 20
- 2023Leave-one-out Distinguishability in Machine Learningcited by 20
- 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
- 2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.