# Revisiting Methods for Finding Influential Examples

arXiv: https://arxiv.org/abs/2111.04683 · HTML: https://mlsanity.com/paper/2111.04683/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 14 of 14)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Auditing Differentially Private Machine Learning: How Private is Private SGD?](https://mlsanity.com/paper/2006.07709/) (2020)
- [Representer Point Selection for Explaining Deep Neural Networks](https://mlsanity.com/paper/1811.09720/) (2018)
- [Learning to Deceive with Attention-Based Explanations](https://mlsanity.com/paper/1909.07913/) (2019)
- [Input Similarity from the Neural Network Perspective](https://mlsanity.com/paper/2102.05262/) (2021)
- [What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play](https://mlsanity.com/paper/1810.09648/) (2018)
- [Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder](https://mlsanity.com/paper/2010.02684/) (2020)
- ["Influence Sketching": Finding Influential Samples In Large-Scale Regressions](https://mlsanity.com/paper/1611.05923/) (2016)

## Built on by (most cited first, 30 of 39)

- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Data Shapley in One Training Run](https://mlsanity.com/paper/2406.11011/) (2024)
- [Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs](https://mlsanity.com/paper/2303.08114/) (2023)
- [Capturing the Temporal Dependence of Training Data Influence](https://mlsanity.com/paper/2412.09538/) (2024)
- [First is Better Than Last for Language Data Influence](https://mlsanity.com/paper/2202.11844/) (2022)
- [ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data](https://mlsanity.com/paper/2205.12600/) (2022)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [Theoretical and Practical Perspectives on what Influence Functions Do](https://mlsanity.com/paper/2305.16971/) (2023)
- [Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits](https://mlsanity.com/paper/2405.03875/) (2024)
- [Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets](https://mlsanity.com/paper/2302.13959/) (2023)
- [Unlearning Traces the Influential Training Data of Language Models](https://mlsanity.com/paper/2401.15241/) (2024)
- [textttdattri: A Library for Efficient Data Attribution](https://mlsanity.com/paper/2410.04555/) (2024)
- [A Bayesian Approach To Analysing Training Data Attribution In Deep Learning](https://mlsanity.com/paper/2305.19765/) (2023)
- [Unifying Corroborative and Contributive Attributions in Large Language Models](https://mlsanity.com/paper/2311.12233/) (2023)
- [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [On Training Data Influence of GPT Models](https://mlsanity.com/paper/2404.07840/) (2024)
- [Gradient-Based Automated Iterative Recovery for Parameter-Efficient Tuning](https://mlsanity.com/paper/2302.06598/) (2023)
- [Efficient Ensembles Improve Training Data Attribution](https://mlsanity.com/paper/2405.17293/) (2024)
- [Coresets from Trajectories: Selecting Data via Correlation of Loss Differences](https://mlsanity.com/paper/2508.20230/) (2025)
- [Z0-Inf: Zeroth Order Approximation for Data Influence](https://mlsanity.com/paper/2510.11832/) (2025)
- [Tracing Privacy Leakage of Language Models to Training Data via Adjusted Influence Functions](https://mlsanity.com/paper/2408.10468/) (2024)
- [Quanda: An Interpretability Toolkit for Training Data Attribution Evaluation and Beyond](https://mlsanity.com/paper/2410.07158/) (2024)
- [Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models](https://mlsanity.com/paper/2308.08774/) (2023)
- [Enhancing High-Energy Particle Physics Collision Analysis through Graph Data Attribution Techniques](https://mlsanity.com/paper/2407.14859/) (2024)
- [MelissaDL x Breed: Towards Data-Efficient On-line Supervised Training of Multi-parametric Surrogates with Active Learning](https://mlsanity.com/paper/2410.05860/) (2024)
- [Finding the Muses: Identifying Coresets through Loss Trajectories](https://mlsanity.com/paper/2503.09721/) (2025)

## Abstract (arXiv)

Several instance-based explainability methods for finding influential training examples for test-time decisions have been proposed recently, including Influence Functions, TraceIn, Representer Point Selection, Grad-Dot, and Grad-Cos. Typically these methods are evaluated using LOO influence (Cook's distance) as a gold standard, or using various heuristics. In this paper, we show that all of the above methods are unstable, i.e., extremely sensitive to initialization, ordering of the training data, and batch size. We suggest that this is a natural consequence of how in the literature, the influence of examples is assumed to be independent of model state and other examples -- and argue it is not. We show that LOO influence and heuristics are, as a result, poor metrics to measure the quality of instance-based explanations, and instead propose to evaluate such explanations by their ability to detect poisoning attacks. Further, we provide a simple, yet effective baseline to improve all of the above methods and show how it leads to very significant improvements on downstream tasks.