A Bayesian Information-Theoretic Approach to Data Attribution
Not on the map. Why?
Abstract (arXiv)
Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a Bayesian information-theoretic problem: subsets are scored by the information loss they induce - the entropy increase at a query when removed. This criterion credits examples for resolving predictive uncertainty rather than label noise. To scale to modern networks, we approximate information loss using a Gaussian Process surrogate built from tangent features. We show this aligns with classical influence scores for single-example attribution while promoting diversity for subsets. For even larger-scale retrieval, we relax to an information-gain objective and add a variance correction for scalable attribution in vector databases. Experiments show competitive performance on counterfactual sensitivity, ground-truth retrieval and coreset selection, showing that our method scales to modern architectures while bridging principled measures with practice.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 4 Apr 2026.
Builds on
27 of 42 references.
References · 27
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2023Studying Large Language Model Generalization with Influence Functionscited by 326
- 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
- 2023TRAK: Attributing Model Behavior at Scalecited by 283
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
- 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
- 2021Scaling Up Influence Functionscited by 132
15 more
- 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
- 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
- 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
- 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 79
- 2019Data Cleansing for Models Trained with SGDcited by 67
- 2017Covariances, Robustness, and Variational Bayescited by 40
- 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 34
- 2020RelatIF: Identifying Explanatory Training Examples via Relative Influencecited by 28
- 2025OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokenscited by 27
- 2023A Bayesian Approach To Analysing Training Data Attribution In Deep Learningcited by 11
- 2024Efficient Sketches for Training Data Attribution and Studying the Loss Landscapecited by 10
- 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
- 2023A Simple and Efficient Baseline for Data Attribution on Imagescited by 5
- 2024Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methodscited by 5
- 2024A Bayesian Approach to Data Point Selectioncited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.