LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution
Not on the map. Why?
Abstract (arXiv)
Training data attribution (TDA) identifies which training examples most influenced a model's prediction. Influence function methods are a theoretically grounded family of TDA methods and exploit gradients. To overcome the scalability challenge arising from gradient computation, the most popular strategy is random projection (e.g., TRAK, LoGRA). However, this still faces two bottlenecks when scaling to large training sets and high-quality attribution: (i) storing and loading projected per-example gradients for all N training examples, where query latency is dominated by I/O; and (ii) forming the D × D inverse Hessian approximation, which costs O(D2) memory. Both bottlenecks scale with the projection dimension D, yet increasing D is necessary for attribution quality -- creating a quality--scalability tradeoff. We introduce LoRIF (Low-Rank Influence Functions), which exploits low-rank structures of gradient to address both bottlenecks. First, we store rank-c factors of projected per-example gradients rather than full matrices, reducing storage and query-time I/O from O(D) to O(c√D) per layer per sample. Second, we use truncated SVD with the Woodbury identity to approximate the inverse Hessian term in an r-dimensional subspace, reducing memory from O(D2) to O(Dr). On models from 0.1B to 70B parameters trained on datasets with millions of examples, LoRIF achieves up to 20× storage reduction and query-time speedup compared to LoGRA, while matching or exceeding its attribution quality. LoRIF makes gradient-based TDA practical at frontier scale.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
34 of 46 references.
References · 34
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
- 2023Studying Large Language Model Generalization with Influence Functionscited by 326
- 2023TRAK: Attributing Model Behavior at Scalecited by 283
- 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
- 2022Datamodels: Predicting Predictions from Training Datacited by 189
- 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
22 more
- 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
- 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
- 2021Scaling Up Influence Functionscited by 132
- 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
- 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
- 2021Input Similarity from the Neural Network Perspectivecited by 79
- 2019Data Cleansing for Models Trained with SGDcited by 67
- 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 53
- 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
- 2021Interactive Label Cleaning with Example-based Explanationscited by 49
- 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
- 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
- 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 42
- 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
- 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
- 2020PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learningcited by 35
- 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
- 2020Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkcited by 20
- 2016"Influence Sketching": Finding Influential Samples In Large-Scale Regressionscited by 17
- 2024Efficient Sketches for Training Data Attribution and Studying the Loss Landscapecited by 10
- 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
- 2025GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projectioncited by 8
Built on by
Citing papers per year: 2026: 3.
3 papers.