# Capturing the Temporal Dependence of Training Data Influence

arXiv: https://arxiv.org/abs/2412.09538 · HTML: https://mlsanity.com/paper/2412.09538/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 76 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [Data Shapley in One Training Run](https://mlsanity.com/paper/2406.11011/) (2024)
- [Revisiting Methods for Finding Influential Examples](https://mlsanity.com/paper/2111.04683/) (2021)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution](https://mlsanity.com/paper/2401.15866/) (2024)
- [Fundamentals of Task-Agnostic Data Valuation](https://mlsanity.com/paper/2208.12354/) (2022)
- [Threshold KNN-Shapley: A Linear-Time and Privacy-Friendly Approach to Data Valuation](https://mlsanity.com/paper/2308.15709/) (2023)
- [Efficient Data Shapley for Weighted Nearest Neighbor Algorithms](https://mlsanity.com/paper/2401.11103/) (2024)
- [Estimation of the Shapley value by ergodic sampling](https://mlsanity.com/paper/1906.05224/) (2019)
- [Private Data Valuation and Fair Payment in Data Marketplaces](https://mlsanity.com/paper/2210.08723/) (2022)
- [A Note on "Towards Efficient Data Valuation Based on the Shapley Value''](https://mlsanity.com/paper/2302.11431/) (2023)
- [A Note on "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms"](https://mlsanity.com/paper/2304.04258/) (2023)
- [TimeInf: Time Series Data Contribution via Influence Functions](https://mlsanity.com/paper/2407.15247/) (2024)
- [GMValuator: Similarity-based Data Valuation for Generative Models](https://mlsanity.com/paper/2304.10701/) (2023)
- [Efficient Ensembles Improve Training Data Attribution](https://mlsanity.com/paper/2405.17293/) (2024)
- [On the Inflation of KNN-Shapley Value](https://mlsanity.com/paper/2405.17489/) (2024)
- [Computational Copyright: Towards A Royalty Model for Music Generative AI](https://mlsanity.com/paper/2312.06646/) (2023)

## Built on by (most cited first, 30 of 35)

- [Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities](https://mlsanity.com/paper/2501.12147/) (2025)
- [OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration](https://mlsanity.com/paper/2602.05400/) (2026)
- [Group-Level Data Selection for Efficient Pretraining](https://mlsanity.com/paper/2502.14709/) (2025)
- [A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning](https://mlsanity.com/paper/2505.19281/) (2025)
- [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025)
- [Enhancing Training Data Attribution with Representational Optimization](https://mlsanity.com/paper/2505.18513/) (2025)
- [GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection](https://mlsanity.com/paper/2505.18976/) (2025)
- [DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models](https://mlsanity.com/paper/2507.09424/) (2025)
- [How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines](https://mlsanity.com/paper/2605.18814/) (2026)
- [Understanding Data Influence with Differential Approximation](https://mlsanity.com/paper/2508.14648/) (2025)
- [OATS: Online Data Augmentation for Time Series Foundation Models](https://mlsanity.com/paper/2601.19040/) (2026)
- [Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems](https://mlsanity.com/paper/2506.00175/) (2025)
- [Newfluence: Boosting Model interpretability and Understanding in High Dimensions](https://mlsanity.com/paper/2507.11895/) (2025)
- [Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs](https://mlsanity.com/paper/2509.25380/) (2025)
- [Layer-Aware Influence for Online Data Valuation Estimation](https://mlsanity.com/paper/2510.16007/) (2025)
- [Unifying and Optimizing Data Values for Selection via Sequential Decision-Making](https://mlsanity.com/paper/2502.04554/) (2025)
- [Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning](https://mlsanity.com/paper/2604.00001/) (2026)
- [AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training](https://mlsanity.com/paper/2606.01635/) (2026)
- [STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations](https://mlsanity.com/paper/2606.05165/) (2026)
- [Do Neural Networks Preserve Case Structure? Case-Based Decomposition, Interpretation, and Decision Consistency](https://mlsanity.com/paper/2607.11347/) (2026)
- [Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW](https://mlsanity.com/paper/2608.19762/) (2026)
- [Exploring Training Data Attribution under Limited Access Constraints](https://mlsanity.com/paper/2509.12581/) (2025)
- [Data-Efficient Training by Evolved Sampling](https://mlsanity.com/paper/2509.23461/) (2025)
- [LLM generation novelty through the lens of semantic similarity](https://mlsanity.com/paper/2510.27313/) (2025)
- [Algorithm-Relative Trajectory Valuation in Policy Gradient Control](https://mlsanity.com/paper/2511.07878/) (2025)
- [GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency](https://mlsanity.com/paper/2603.00031/) (2026)
- [Why Does It Look There? Structured Explanations for Image Classification](https://mlsanity.com/paper/2603.10234/) (2026)
- [Data Attribution in Adaptive Learning](https://mlsanity.com/paper/2604.04892/) (2026)
- [Efficient Dataset Selection for Continual Adaptation of Generative Recommenders](https://mlsanity.com/paper/2604.07739/) (2026)
- [Dynamic Important Example Mining for Reinforcement Finetuning](https://mlsanity.com/paper/2608.29252/) (2026)

## Abstract (arXiv)

Traditional data influence estimation methods, like influence function, assume that learning algorithms are permutation-invariant with respect to training data. However, modern training paradigms, especially for foundation models using stochastic algorithms and multi-stage curricula, are sensitive to data ordering, thus violating this assumption. This mismatch renders influence functions inadequate for answering a critical question in machine learning: How can we capture the dependence of data influence on the optimization trajectory during training? To address this gap, we formalize the concept of trajectory-specific leave-one-out (LOO) influence, which quantifies the impact of removing a data point from a specific iteration during training, accounting for the exact sequence of data encountered and the model's optimization trajectory. However, exactly evaluating the trajectory-specific LOO presents a significant computational challenge. To address this, we propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO. Specifically, we compute a training data embedding that encapsulates the cumulative interactions between data and the evolving model parameters. The LOO can then be efficiently approximated through a simple dot-product between the data value embedding and the gradient of the given test data. As data value embedding captures training data ordering, it offers valuable insights into model training dynamics. In particular, we uncover distinct phases of data influence, revealing that data points in the early and late stages of training exert a greater impact on the final model. These insights translate into actionable strategies for managing the computational overhead of data selection by strategically timing the selection process, potentially opening new avenues in data curation research.