# Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining

arXiv: https://arxiv.org/abs/2505.24261 · HTML: https://mlsanity.com/paper/2505.24261/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Sketching as a Tool for Numerical Linear Algebra](https://mlsanity.com/paper/1411.4357/) (2014)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Towards Efficient Data Valuation Based on the Shapley Value](https://mlsanity.com/paper/1902.10275/) (2019)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [Data Shapley in One Training Run](https://mlsanity.com/paper/2406.11011/) (2024)
- [OpenDataVal: a Unified Benchmark for Data Valuation](https://mlsanity.com/paper/2306.10577/) (2023)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime](https://mlsanity.com/paper/2112.08297/) (2021)
- [textttdattri: A Library for Efficient Data Attribution](https://mlsanity.com/paper/2410.04555/) (2024)
- [Revisiting inverse Hessian vector products for calculating influence functions](https://mlsanity.com/paper/2409.17357/) (2024)

## Built on by (most cited first, 4 of 4)

- [A Unified Theory of Random Projection for Influence Functions](https://mlsanity.com/paper/2602.10449/) (2026)
- [Challenges in Enabling Private Data Valuation](https://mlsanity.com/paper/2603.00342/) (2026)
- [Exploring Training Data Attribution under Limited Access Constraints](https://mlsanity.com/paper/2509.12581/) (2025)
- [On the Fragility of Data Attribution When Learning Is Distributed](https://mlsanity.com/paper/2605.15520/) (2026)

## Abstract (arXiv)

Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric applications in modern AI. Despite a recent surge of new methods developed in this space, the impact of hyperparameter tuning in these methods remains under-explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. However, unlike typical machine learning algorithms -- whose hyperparameters can be tuned using computationally-cheap validation metrics -- evaluating data attribution performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses a critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparameter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies a fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion on hyperparameter selection in future method development.