# Understanding Data Influence with Differential Approximation arXiv: https://arxiv.org/abs/2508.14648 ยท HTML: https://mlsanity.com/paper/2508.14648/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 60 of its 85 references that are in the map. - First posted on arXiv 20 Aug 2025. ## Builds on (most cited first, 30 of 60) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017) - [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015) - [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021) - [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017) - [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020) - [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019) - [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020) - [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024) - [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022) - [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020) - [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020) - [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018) - [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023) - [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014) - [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023) - [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014) - [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017) - [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020) ## Built on by (most cited first, 3 of 3) - [TRUE: A Trustworthy Unified Explanation Framework for Large Language Model Reasoning](https://mlsanity.com/paper/2602.18905/) (2026) - [Contour-Native Bridge Defect Detection and Compact Digital Archiving with Frequency-Supervised Fourier Contours](https://mlsanity.com/paper/2605.08781/) (2026) - [Dynamic Important Example Mining for Reinforcement Finetuning](https://mlsanity.com/paper/2608.29252/) (2026) ## Abstract (arXiv) Data plays a pivotal role in the groundbreaking advancements in artificial intelligence. The quantitative analysis of data significantly contributes to model training, enhancing both the efficiency and quality of data utilization. However, existing data analysis tools often lag in accuracy. For instance, many of these tools even assume that the loss function of neural networks is convex. These limitations make it challenging to implement current methods effectively. In this paper, we introduce a new formulation to approximate a sample's influence by accumulating the differences in influence between consecutive learning steps, which we term Diff-In. Specifically, we formulate the sample-wise influence as the cumulative sum of its changes/differences across successive training iterations. By employing second-order approximations, we approximate these difference terms with high accuracy while eliminating the need for model convexity required by existing methods. Despite being a second-order method, Diff-In maintains computational complexity comparable to that of first-order methods and remains scalable. This efficiency is achieved by computing the product of the Hessian and gradient, which can be efficiently approximated using finite differences of first-order gradients. We assess the approximation accuracy of Diff-In both theoretically and empirically. Our theoretical analysis demonstrates that Diff-In achieves significantly lower approximation error compared to existing influence estimators. Extensive experiments further confirm its superior performance across multiple benchmark datasets in three data-centric tasks: data cleaning, data deletion, and coreset selection. Notably, our experiments on data pruning for large-scale vision-language pre-training show that Diff-In can scale to millions of data points and outperforms strong baselines.