mlsanityData to 14 Oct 2026

Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime

arXiv 2112.08297 ↗First posted 15 Dec 2021cs.LG cs.AI stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

Understanding the black-box prediction for neural networks is challenging. To achieve this, early studies have designed influence function (IF) to measure the effect of removing a single training point on neural networks. However, the classic implicit Hessian-vector product (IHVP) method for calculating IF is fragile, and theoretical analysis of IF in the context of neural networks is still lacking. To this end, we utilize the neural tangent kernel (NTK) theory to calculate IF for the neural network trained with regularized mean-square loss, and prove that the approximation error can be arbitrarily small when the width is sufficiently large for two-layer ReLU networks. We analyze the error bound for the classic IHVP method in the over-parameterized regime to understand when and why it fails or not. In detail, our theoretical analysis reveals that (1) the accuracy of IHVP depends on the regularization term, and is pretty low under weak regularization; (2) the accuracy of IHVP has a significant correlation with the probability density of corresponding training points. We further borrow the theory from NTK to understand the IFs better, including quantifying the complexity for influential samples and depicting the variation of IFs during the training dynamics. Numerical experiments on real-world data confirm our theoretical results and demonstrate our findings.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

23 of 32 references.

References · 23

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  9. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  10. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  11. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  12. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 161
11 more
  1. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  2. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  3. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 147
  4. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  5. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  6. 2019Repairing without Retraining: Avoiding Disparate Impact with Counterfactual Distributionscited by 48
  7. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  8. 2016Diverse Neural Network Learns True Target Functionscited by 33
  9. 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 28
  10. 2018Data Dropout: Optimizing Training Data for Convolutional Neural Networkscited by 23
  11. 2021Understanding Instance-based Interpretability of Variational Auto-Encoderscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 7; 2024: 6; 2025: 6; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  2. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  3. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  4. 2024Machine Unlearning Fails to Remove Data Poisoning Attackscited by 37
  5. 2023Data Valuation Without Training of a Modelcited by 35
  6. 2022Identifying a Training-Set Attack's Target Using Renormalized Influence Estimationcited by 32
  7. 2024Reconstruction Attacks on Machine Unlearning: Simple Models are Vulnerablecited by 25
  8. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  9. 2023Regularizing Second-Order Influences for Continual Learningcited by 21
  10. 2022On the Trade-Off between Actionable Explanations and the Right to be Forgottencited by 20
  11. 2023A Bayesian Approach To Analysing Training Data Attribution In Deep Learningcited by 11
  12. 2022Generalization Through The Lens Of Leave-One-Out Errorcited by 10
14 more
  1. 2023Sample based Explanations via Generalized Representerscited by 10
  2. 2024Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methodscited by 5
  3. 2025Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retrainingcited by 4
  4. 2026The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samplescited by 4
  5. 2025UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptationcited by 2
  6. 2024OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attackcited by 1
  7. 2024PINNfluence: Interpreting PINNs through Influence Functionscited by 1
  8. 2024Where Did Your Model Learn That? Label-free Influence for Self-supervised Learningcited by 1
  9. 2025Flexible Concept Bottleneck Modelcited by 1
  10. 2023On the Trade-offs between Adversarial Robustness and Actionable Explanationscited by 0
  11. 2025Interpretable Feature Interaction via Statistical Self-supervised Learning on Tabular Datacited by 0
  12. 2025A Comparative Analysis of Influence Signals for Data Debuggingcited by 0
  13. 2025f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomnesscited by 0
  14. 2026Linearized Attention Cannot Enter the Kernel Regime at Any Practical Widthcited by 0