mlsanityData to 14 Oct 2026

The Memory Perturbation Equation: Understanding Model's Sensitivity to Data

arXiv 2310.19273 ↗First posted 30 Oct 2023cs.LG cs.AI stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

32 of 57 references.

References · 32

  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  2. 2017A Closer Look at Memorization in Deep Networkscited by 942
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  5. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  7. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  8. 2013Gaussian Processes for Big Datacited by 383
  9. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  11. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  12. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
20 more
  1. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 147
  2. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  3. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  4. 2021The Bayesian Learning Rulecited by 99
  5. 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
  6. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  7. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  8. 2019Data Cleansing for Models Trained with SGDcited by 67
  9. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  10. 2019Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networkscited by 45
  11. 2019One Epoch Is All You Needcited by 41
  12. 2017Covariances, Robustness, and Variational Bayescited by 40
  13. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  14. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  15. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  16. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  17. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 21
  18. 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
  19. 2022Generalization Through The Lens Of Leave-One-Out Errorcited by 10
  20. 2023Variational Bayes Made Easycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  2. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 34
  3. 2025Generative Uncertainty in Diffusion Modelscited by 23
  4. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  5. 2024Explainable Natural Language Processing for Corporate Sustainability Analysiscited by 9
  6. 2025Knowledge Adaptation as Posterior Correctioncited by 6
  7. 2024Revisiting the robustness of post-hoc interpretability methodscited by 5
  8. 2024Variational Low-Rank Adaptation Using IVONcited by 5
  9. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  10. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 3
  11. 2025DICE: Data Influence Cascade in Decentralized Learningcited by 2
  12. 2025Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influencecited by 1
4 more
  1. 2024Set-Valued Sensitivity Analysis of Deep Neural Networkscited by 0
  2. 2025Variational Learning Induces Adaptive Label Smoothingcited by 0
  3. 2026Joint Model and Data Sparsification via the Marginal Likelihoodcited by 0
  4. 2026Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behaviorcited by 0