mlsanityData to 14 Oct 2026

GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection

arXiv 2505.18976 ↗First posted 25 May 2025cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Gradient-based data attribution methods, such as influence functions, are critical for understanding the impact of individual training samples without requiring repeated model retraining. However, their scalability is often limited by the high computational and memory costs associated with per-sample gradient computation. In this work, we propose GraSS, a novel gradient compression algorithm and its variants FactGraSS for linear layers specifically, that explicitly leverage the inherent sparsity of per-sample gradients to achieve sub-linear space and time complexity. Extensive experiments demonstrate the effectiveness of our approach, achieving substantial speedups while preserving data influence fidelity. In particular, FactGraSS achieves up to 165% faster throughput on billion-scale models compared to the previous state-of-the-art baselines. Our code is publicly available at https://github.com/TRAIS-Lab/GraSS.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

32 of 52 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  8. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  9. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  10. 2014Sketching as a Tool for Numerical Linear Algebracited by 288
  11. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  12. 2017Sparse Communication for Distributed Gradient Descentcited by 249
20 more
  1. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
  2. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  3. 2021Scaling Up Influence Functionscited by 132
  4. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  5. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 109
  6. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
  7. 2021Input Similarity from the Neural Network Perspectivecited by 79
  8. 2019Data Cleansing for Models Trained with SGDcited by 67
  9. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  10. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  11. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  12. 2019Fast and Accurate Network Embeddings via Very Sparse Random Projectioncited by 29
  13. 2018Billion-scale Network Embedding with Iterative Random Projectioncited by 25
  14. 2024Token-wise Influential Training Data Retrieval for Large Language Modelscited by 20
  15. 2024textttdattri: A Library for Efficient Data Attributioncited by 18
  16. 2016"Influence Sketching": Finding Influential Samples In Large-Scale Regressionscited by 17
  17. 2023OPORP: One Permutation + One Random Projectioncited by 12
  18. 2016Lecture Notes on Randomized Linear Algebracited by 11
  19. 2024Efficient Sketches for Training Data Attribution and Studying the Loss Landscapecited by 10
  20. 2022The Fast Johnson-Lindenstrauss Transform is Even Fastercited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Step-resolved data attribution for looped transformerscited by 5
  2. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 3
  3. 2026A Unified Theory of Random Projection for Influence Functionscited by 1
  4. 2026STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbationscited by 1
  5. 2026DRIFT: Data Selection for LLM Instruction Tuning via On-Policy Attributioncited by 1
  6. 2026EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptationcited by 0
  7. 2026Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projectioncited by 0
  8. 2026dattri-LLM: A Unified and Efficient Library for Training Data Attribution at LLM Scalecited by 0