# Multi-Stage Influence Function

arXiv: https://arxiv.org/abs/2007.09081 · HTML: https://mlsanity.com/paper/2007.09081/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2020.

## Built on by (most cited first, 16 of 16)

- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data](https://mlsanity.com/paper/2205.12600/) (2022)
- [Editable Concept Bottleneck Models](https://mlsanity.com/paper/2405.15476/) (2024)
- [Understanding Instance-based Interpretability of Variational Auto-Encoders](https://mlsanity.com/paper/2105.14203/) (2021)
- [Regularizing Second-Order Influences for Continual Learning](https://mlsanity.com/paper/2304.10177/) (2023)
- [Interpreting Unfairness in Graph Neural Networks via Training Node Attribution](https://mlsanity.com/paper/2211.14383/) (2022)
- [Understanding Programmatic Weak Supervision via Source-aware Influence Function](https://mlsanity.com/paper/2205.12879/) (2022)
- [Dissecting Representation Misalignment in Contrastive Learning via Influence Function](https://mlsanity.com/paper/2411.11667/) (2024)
- [Attributing Data for Sharpness-Aware Minimization](https://mlsanity.com/paper/2507.04059/) (2025)
- [Controllable Concept Bottleneck Models](https://mlsanity.com/paper/2601.00451/) (2026)
- [Additional Positive Enables Better Representation Learning for Medical Images](https://mlsanity.com/paper/2306.00112/) (2023)
- [Channel Matters: Estimating Channel Influence for Multivariate Time Series](https://mlsanity.com/paper/2408.14763/) (2024)
- [Scalable Multi-Stage Influence Function for Large Language Models via Eigenvalue-Corrected Kronecker-Factored Parameterization](https://mlsanity.com/paper/2505.05017/) (2025)
- [Learning to Fast Unrank in Collaborative Filtering Recommendation](https://mlsanity.com/paper/2511.06803/) (2025)

## Abstract (arXiv)

Multi-stage training and knowledge transfer, from a large-scale pretraining task to various finetuning tasks, have revolutionized natural language processing and computer vision resulting in state-of-the-art performance improvements. In this paper, we develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task. The proposed multi-stage influence function generalizes the original influence function for a single model in (Koh & Liang, 2017), thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks. We test our proposed method in various experiments to show its effectiveness and potential applications.