mlsanityData to 7 Oct 2026

Estimating informativeness of samples with Smooth Unique Information

arXiv 2101.06640 ↗First posted 17 Jan 2021cs.LG stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

We define a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the final weights and how much it informs the function computed by the weights. Though related, we show that these quantities have a qualitatively different behavior. We give efficient approximations of these quantities using a linearized network and demonstrate empirically that the approximation is accurate for real-world architectures, such as pre-trained ResNets. We apply these measures to several problems, such as dataset summarization, analysis of under-sampled classes, comparison of informativeness of different data sources, and detection of adversarial and corrupted examples. Our work generalizes existing frameworks but enjoys better computational properties for heavily over-parametrized models, which makes it possible to apply it to real-world networks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 17 Jan 2021.

Builds on

28 of 47 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  8. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
16 more
  1. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  2. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  3. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  4. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
  5. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 143
  6. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  7. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  8. 2019Data Valuation using Reinforcement Learningcited by 115
  9. 2015Algorithmic Stability for Adaptive Data Analysiscited by 88
  10. 2018How Complex is your classification problem? A survey on measuring classification complexitycited by 66
  11. 2019Where is the Information in a Deep Neural Network?cited by 40
  12. 2020Gradients as Features for Deep Representation Learningcited by 30
  13. 2017Calibrating Noise to Variance in Adaptive Data Analysiscited by 28
  14. 2020Predicting Training Time Without Trainingcited by 22
  15. 2019iCassava 2019 Fine-Grained Visual Categorization Challengecited by 17
  16. 2019Information in Infinite Ensembles of Infinitely-Wide Neural Networkscited by 14

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 7; 2023: 4; 2024: 0; 2025: 1.

15 papers.

Sort:

  1. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  2. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  3. 2021Survey: Leakage and Privacy at Inference Timecited by 44
  4. 2023Taming AI Bots: Controllability of Neural States in Large Language Modelscited by 22
  5. 2021Understanding Instance-based Interpretability of Variational Auto-Encoderscited by 21
  6. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  7. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  8. 2023AI Model Disgorgement: Methods and Choicescited by 9
  9. 2022On Leave-One-Out Conditional Mutual Information For Generalizationcited by 8
  10. 2022Measuring Unintended Memorisation of Unique Private Features in Neural Networkscited by 5
  11. 2022Quantifying Feature Contributions to Overall Disparity Using Information Theorycited by 2
  12. 2022Towards Differential Relational Privacy and its use in Question Answeringcited by 1
3 more
  1. 2022Unintended memorisation of unique features in neural networkscited by 1
  2. 2023SoK: Memorisation in machine learningcited by 1
  3. 2025Laplace Sample Information: Data Informativeness Through a Bayesian Lenscited by 1