# Scaling Up Influence Functions

arXiv: https://arxiv.org/abs/2112.03052 · HTML: https://mlsanity.com/paper/2112.03052/

## Key facts

- Cited by 132 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Sketching as a Tool for Numerical Linear Algebra](https://mlsanity.com/paper/1411.4357/) (2014)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Representer Point Selection for Explaining Deep Neural Networks](https://mlsanity.com/paper/1811.09720/) (2018)
- [Mesh-TensorFlow: Deep Learning for Supercomputers](https://mlsanity.com/paper/1811.02084/) (2018)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging](https://mlsanity.com/paper/2012.15781/) (2020)
- [Understanding the Origins of Bias in Word Embeddings](https://mlsanity.com/paper/1810.03611/) (2018)
- [Data Dropout: Optimizing Training Data for Convolutional Neural Networks](https://mlsanity.com/paper/1809.00193/) (2018)
- ["Influence Sketching": Finding Influential Samples In Large-Scale Regressions](https://mlsanity.com/paper/1611.05923/) (2016)
- [Multi-Stage Influence Function](https://mlsanity.com/paper/2007.09081/) (2020)
- [Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits](https://mlsanity.com/paper/2110.06997/) (2021)
- [A Fast, Compact, Accurate Model for Language Identification of Codemixed Text](https://mlsanity.com/paper/1810.04142/) (2018)
- [Microsoft's Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data](https://mlsanity.com/paper/1809.00196/) (2018)

## Built on by (most cited first, 30 of 132)

- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models](https://mlsanity.com/paper/2310.00902/) (2023)
- [MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models](https://mlsanity.com/paper/2406.06046/) (2024)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs](https://mlsanity.com/paper/2303.08114/) (2023)
- [Intriguing Properties of Data Attribution on Diffusion Models](https://mlsanity.com/paper/2311.00500/) (2023)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Evaluating Data Attribution for Text-to-Image Models](https://mlsanity.com/paper/2306.09345/) (2023)
- [Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning](https://mlsanity.com/paper/2403.07362/) (2024)
- [The Journey, Not the Destination: How Data Guides Diffusion Models](https://mlsanity.com/paper/2312.06205/) (2023)
- [Scalable Influence and Fact Tracing for Large Language Model Pretraining](https://mlsanity.com/paper/2410.17413/) (2024)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Theoretical and Practical Perspectives on what Influence Functions Do](https://mlsanity.com/paper/2305.16971/) (2023)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refitting](https://mlsanity.com/paper/2212.06803/) (2022)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [Data Attribution for Text-to-Image Models by Unlearning Synthesized Images](https://mlsanity.com/paper/2406.09408/) (2024)
- [Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models](https://mlsanity.com/paper/2410.18639/) (2024)
- [Towards Tracing Factual Knowledge in Language Models Back to the Training Data](https://mlsanity.com/paper/2205.11482/) (2022)
- [Regularizing Second-Order Influences for Continual Learning](https://mlsanity.com/paper/2304.10177/) (2023)
- [Membership Inference Attack Using Self Influence Functions](https://mlsanity.com/paper/2205.13680/) (2022)
- [Do Influence Functions Work on Large Language Models?](https://mlsanity.com/paper/2409.19998/) (2024)
- [Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets](https://mlsanity.com/paper/2302.13959/) (2023)
- [textttdattri: A Library for Efficient Data Attribution](https://mlsanity.com/paper/2410.04555/) (2024)
- [Natural Example-Based Explainability: a Survey](https://mlsanity.com/paper/2309.03234/) (2023)
- [Contrastive Error Attribution for Finetuned Language Models](https://mlsanity.com/paper/2212.10722/) (2022)
- [LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions](https://mlsanity.com/paper/2505.23811/) (2025)

## Abstract (arXiv)

We address efficient calculation of influence functions for tracking predictions back to the training data. We propose and analyze a new approach to speeding up the inverse Hessian calculation based on Arnoldi iteration. With this improvement, we achieve, to the best of our knowledge, the first successful implementation of influence functions that scales to full-size (language and vision) Transformer models with several hundreds of millions of parameters. We evaluate our approach on image classification and sequence-to-sequence tasks with tens to a hundred of millions of training examples. Our code will be available at https://github.com/google-research/jax-influence.