# Theoretical and Practical Perspectives on what Influence Functions Do arXiv: https://arxiv.org/abs/2305.16971 ยท HTML: https://mlsanity.com/paper/2305.16971/ ## Key facts - Cited by 29 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 19 references that are in the map. - First posted on arXiv 26 May 2023. ## Builds on (most cited first, 15 of 15) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017) - [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020) - [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021) - [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019) - [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014) - [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020) - [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016) - [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019) - [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022) - [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021) - [FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging](https://mlsanity.com/paper/2012.15781/) (2020) - [Input Similarity from the Neural Network Perspective](https://mlsanity.com/paper/2102.05262/) (2021) - [Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs](https://mlsanity.com/paper/2303.08114/) (2023) - [Revisiting Methods for Finding Influential Examples](https://mlsanity.com/paper/2111.04683/) (2021) ## Built on by (most cited first, 29 of 29) - [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024) - [Causal Estimation of Memorisation Profiles](https://mlsanity.com/paper/2406.04327/) (2024) - [Unlearning Traces the Influential Training Data of Language Models](https://mlsanity.com/paper/2401.15241/) (2024) - [LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions](https://mlsanity.com/paper/2505.23811/) (2025) - [Unveiling Privacy, Memorization, and Input Curvature Links](https://mlsanity.com/paper/2402.18726/) (2024) - [Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models](https://mlsanity.com/paper/2405.03869/) (2024) - [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025) - [Enhancing Training Data Attribution with Representational Optimization](https://mlsanity.com/paper/2505.18513/) (2025) - [Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods](https://mlsanity.com/paper/2412.03906/) (2024) - [Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics](https://mlsanity.com/paper/2508.11017/) (2025) - [Train on Validation (ToV): Fast data selection with applications to fine-tuning](https://mlsanity.com/paper/2510.00386/) (2025) - [Z0-Inf: Zeroth Order Approximation for Data Influence](https://mlsanity.com/paper/2510.11832/) (2025) - [Attributing Data for Sharpness-Aware Minimization](https://mlsanity.com/paper/2507.04059/) (2025) - [Newfluence: Boosting Model interpretability and Understanding in High Dimensions](https://mlsanity.com/paper/2507.11895/) (2025) - [Testing Most Influential Sets](https://mlsanity.com/paper/2510.20372/) (2025) - [SoK: Memorisation in machine learning](https://mlsanity.com/paper/2311.03075/) (2023) - [PINNfluence: Interpreting PINNs through Influence Functions](https://mlsanity.com/paper/2409.08958/) (2024) - [Laplace Sample Information: Data Informativeness Through a Bayesian Lens](https://mlsanity.com/paper/2505.15303/) (2025) - [PARIS: Pruning Algorithm via the Representer theorem for Imbalanced Scenarios](https://mlsanity.com/paper/2512.06950/) (2025) - [STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations](https://mlsanity.com/paper/2606.05165/) (2026) - [LayerMatch: Do Pseudo-labels Benefit All Layers?](https://mlsanity.com/paper/2406.14207/) (2024) - [A Comparative Analysis of Influence Signals for Data Debugging](https://mlsanity.com/paper/2506.11584/) (2025) - [Targeted Fine-Tuning of DNN-Based Receivers via Influence Functions](https://mlsanity.com/paper/2509.15950/) (2025) - [What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysis](https://mlsanity.com/paper/2510.03950/) (2025) - [f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomness](https://mlsanity.com/paper/2510.10510/) (2025) - [Data Attribution in Adaptive Learning](https://mlsanity.com/paper/2604.04892/) (2026) - [Data Attribution in Large Language Models via Bidirectional Gradient Optimization](https://mlsanity.com/paper/2606.04928/) (2026) - [Validity Threats for Foundation Model Research](https://mlsanity.com/paper/2606.05029/) (2026) - [Groups Don't Move as One: A Bregman Clustering View of Linearized Group Influence](https://mlsanity.com/paper/2610.14809/) (2026) ## Abstract (arXiv) Influence functions (IF) have been seen as a technique for explaining model predictions through the lens of the training data. Their utility is assumed to be in identifying training examples "responsible" for a prediction so that, for example, correcting a prediction is possible by intervening on those examples (removing or editing them) and retraining the model. However, recent empirical studies have shown that the existing methods of estimating IF predict the leave-one-out-and-retrain effect poorly. In order to understand the mismatch between the theoretical promise and the practical results, we analyse five assumptions made by IF methods which are problematic for modern-scale deep neural networks and which concern convexity, numeric stability, training trajectory and parameter divergence. This allows us to clarify what can be expected theoretically from IF. We show that while most assumptions can be addressed successfully, the parameter divergence poses a clear limitation on the predictive power of IF: influence fades over training time even with deterministic training. We illustrate this theoretical result with BERT and ResNet models. Another conclusion from the theoretical analysis is that IF are still useful for model debugging and correcting even though some of the assumptions made in prior work do not hold: using natural language processing and computer vision tasks, we verify that mis-predictions can be successfully corrected by taking only a few fine-tuning steps on influential examples.