mlsanityData to 14 Oct 2026

Efficient Sketches for Training Data Attribution and Studying the Loss Landscape

arXiv 2402.03994 ↗First posted 6 Feb 2024cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

The study of modern machine learning models often necessitates storing vast quantities of gradients or Hessian vector products (HVPs). Traditional sketching methods struggle to scale under these memory constraints. We present a novel framework for scalable gradient and HVP sketching, tailored for modern hardware. We provide theoretical guarantees and demonstrate the power of our methods in applications like training data attribution, Hessian spectrum analysis, and intrinsic dimension computation for pre-trained language models. Our work sheds new light on the behavior of pre-trained language models, challenging assumptions about their intrinsic dimensionality and Hessian properties.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

20 of 33 references.

References · 20

  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  3. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  5. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  6. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  8. 2014Sketching as a Tool for Numerical Linear Algebracited by 288
  9. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  11. 2021Scaling Up Influence Functionscited by 132
  12. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
8 more
  1. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 109
  2. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  3. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 74
  4. 2022First is Better Than Last for Language Data Influencecited by 34
  5. 2023Spectral alignment of stochastic gradient descent for high-dimensional classification taskscited by 13
  6. 2016Lecture Notes on Randomized Linear Algebracited by 11
  7. 2023Optimal Eigenvalue Approximation via Sketchingcited by 9
  8. 2016Fast nonlinear embeddings via structured matricescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 7.

10 papers.

  1. 2024Revisiting inverse Hessian vector products for calculating influence functionscited by 10
  2. 2025GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projectioncited by 8
  3. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 3
  4. 2024LossVal: Efficient Data Valuation for Neural Networkscited by 1
  5. 2026A Unified Theory of Random Projection for Influence Functionscited by 1
  6. 2026A Bayesian Information-Theoretic Approach to Data Attributioncited by 1
  7. 2026Do Neural Networks Preserve Case Structure? Case-Based Decomposition, Interpretation, and Decision Consistencycited by 1
  8. 2026On the Fragility of Data Attribution When Learning Is Distributedcited by 0
  9. 2026Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projectioncited by 0
  10. 2026Certified Approximation for Interpretable Representer Landmarkscited by 0