# MAGIC: Near-Optimal Data Attribution for Deep Learning

arXiv: https://arxiv.org/abs/2504.16430 · HTML: https://mlsanity.com/paper/2504.16430/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs](https://mlsanity.com/paper/2303.08114/) (2023)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [Optimizing ML Training with Metagradient Descent](https://mlsanity.com/paper/2503.13751/) (2025)
- [On the Variance of Neural Network Training with respect to Test Sets and Distributions](https://mlsanity.com/paper/2304.01910/) (2023)

## Built on by (most cited first, 22 of 22)

- [CUPID: Curating Data your Robot Loves with Influence Functions](https://mlsanity.com/paper/2506.19121/) (2025)
- [Subliminal Effects in Your Data: A General Mechanism via Log-Linearity](https://mlsanity.com/paper/2602.04863/) (2026)
- [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025)
- [Train Once, Answer All: Many Pretraining Experiments for the Cost of One](https://mlsanity.com/paper/2509.23383/) (2025)
- [Infusion: Shaping Model Behavior by Editing Training Data via Influence Functions](https://mlsanity.com/paper/2602.09987/) (2026)
- [Efficient Estimation of Kernel Surrogate Models for Task Attribution](https://mlsanity.com/paper/2602.03783/) (2026)
- [How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines](https://mlsanity.com/paper/2605.18814/) (2026)
- [Distributional Training Data Attribution: What do Influence Functions Sample?](https://mlsanity.com/paper/2506.12965/) (2025)
- [Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions](https://mlsanity.com/paper/2509.23437/) (2025)
- [In-Run Data Shapley for Adam Optimizer](https://mlsanity.com/paper/2602.00329/) (2026)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [Bergson: An Open Source Library for Data Attribution](https://mlsanity.com/paper/2606.11660/) (2026)
- [How to sketch a learning algorithm](https://mlsanity.com/paper/2604.07328/) (2026)
- [Prototype Language Models](https://mlsanity.com/paper/2607.00510/) (2026)
- [Challenges in Enabling Private Data Valuation](https://mlsanity.com/paper/2603.00342/) (2026)
- [Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learning](https://mlsanity.com/paper/2603.00811/) (2026)
- [Small edits, large models: How Wikipedia advocacy shapes LLM values](https://mlsanity.com/paper/2606.24890/) (2026)
- [Assert, don't describe: Linguistic features that shift LLM reasoning about animal welfare](https://mlsanity.com/paper/2606.26104/) (2026)
- [Data Attribution via Sketched Metadifferentiation](https://mlsanity.com/paper/2609.15044/) (2026)
- [Scalable Attribution and Control of Model Behavior During Training](https://mlsanity.com/paper/2609.33667/) (2026)
- [dattri-LLM: A Unified and Efficient Library for Training Data Attribution at LLM Scale](https://mlsanity.com/paper/2609.38767/) (2026)
- [Framework for an AI-Ready ARPES Community Database](https://mlsanity.com/paper/2610.15236/) (2026)

## Abstract (arXiv)

The goal of predictive data attribution is to estimate how adding or removing a given set of training datapoints will affect model predictions. In convex settings, this goal is straightforward (i.e., via the infinitesimal jackknife). In large-scale (non-convex) settings, however, existing methods are far less successful -- current methods' estimates often only weakly correlate with ground truth. In this work, we present a new data attribution method (MAGIC) that combines classical methods and recent advances in metadifferentiation to (nearly) optimally estimate the effect of adding or removing training data on model predictions.