mlsanityData to 7 Oct 2026

TRAK: Attributing Model Behavior at Scale

arXiv 2303.14186 ↗First posted 24 Mar 2023stat.ML cs.LGcited by 273

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The goal of data attribution is to trace model predictions back to training data. Despite a long line of work towards this goal, existing approaches to data attribution tend to force users to choose between computational tractability and efficacy. That is, computationally tractable methods can struggle with accurately attributing model predictions in non-convex settings (e.g., in the context of deep neural networks), while methods that are effective in such regimes require training thousands of models, which makes them impractical for large models or datasets. In this work, we introduce TRAK (Tracing with the Randomly-projected After Kernel), a data attribution method that is both effective and computationally tractable for large-scale, differentiable models. In particular, by leveraging only a handful of trained models, TRAK can match the performance of attribution methods that require training thousands of models. We demonstrate the utility of TRAK across various modalities and scales: image classifiers trained on ImageNet, vision-language models (CLIP), and language models (BERT and mT5). We provide code for using TRAK (and reproducing our work) at https://github.com/MadryLab/trak .

Key facts. Cited by 273 papers in the map (data through 7 Oct 2026). Builds on 80 of its 111 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

80 of 111 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  11. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  12. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  10. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  11. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  14. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  15. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  16. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  17. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  18. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  19. 2014New insights and perspectives on the natural gradient methodcited by 435
  20. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  21. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 279
  22. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  23. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  24. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  25. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  26. 2016What makes ImageNet good for transfer learning?cited by 192
  27. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  28. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  29. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  30. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  31. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  32. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  33. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  34. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  35. 2021Scaling Up Influence Functionscited by 130
  36. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  37. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  38. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 79; 2025: 88; 2026: 81.

273 papers.

Sort:

By area:

  1. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  2. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  3. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  4. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  5. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  6. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  7. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  8. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  9. 2024Data Shapley in One Training Runcited by 76
  10. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  11. 2023Describing Differences in Image Sets with Natural Languagecited by 51
  12. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
38 more
  1. 2023Evaluating Data Attribution for Text-to-Image Modelscited by 45
  2. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  3. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  4. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
  5. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  6. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  7. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
  8. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  9. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  10. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  11. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  12. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  13. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 31
  14. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  15. 2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
  16. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  17. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 28
  18. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 25
  19. 2024Data Attribution for Text-to-Image Models by Unlearning Synthesized Imagescited by 22
  20. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 22
  21. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
  22. 2024Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Modelscited by 21
  23. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  24. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  25. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20
  26. 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
  27. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  28. 2024Do Influence Functions Work on Large Language Models?cited by 20
  29. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 20
  30. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  31. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  32. 2024Unlearning Traces the Influential Training Data of Language Modelscited by 18
  33. 2024An Economic Solution to Copyright Challenges of Generative AIcited by 18
  34. 2024textttdattri: A Library for Efficient Data Attributioncited by 18
  35. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  36. 2023Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sourcescited by 16
  37. 2024The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passescited by 16
  38. 2024The Pitfalls of Memorization: When Memorization Hurts Generalizationcited by 16

50 most cited of 273 All 273 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Explaining the Reasoning of Large Language Models Using Attribution Graphs
  2. 2026A Good Initialization is All You Need for Faithful Visual Attribution
  3. 2026From Behavior to Provenance: Attributing Tabular Foundation Models to Synthetic Pretraining Data
  4. 2024Detecting Origin Attribution for Text-to-Image Diffusion Models
  5. 2026Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning
  6. 2026From internal representations to model improvement through prediction errors
  7. 2026PhaseWin: An Efficient Search Algorithm for Faithful Visual Attribution
  8. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs