mlsanityData to 14 Oct 2026

Influence Dynamics and Stagewise Data Attribution

arXiv 2510.12071 ↗First posted 14 Oct 2025cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Current training data attribution (TDA) methods treat the influence one sample has on another as static, but neural networks learn in distinct stages that exhibit changing patterns of influence. In this work, we introduce a framework for stagewise data attribution grounded in singular learning theory. We predict that influence can change non-monotonically, including sign flips and sharp peaks at developmental transitions. We first validate these predictions analytically and empirically in a toy model, showing that dynamic shifts in influence directly map to the model's progressive learning of a semantic hierarchy. Finally, we demonstrate these phenomena at scale in language models, where token-level influence changes align with known developmental stages.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

31 of 56 references.

References · 31

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2018On the Spectral Bias of Neural Networkscited by 1,341
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  9. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  10. 2017Automated Curriculum Learning for Neural Networkscited by 257
  11. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
19 more
  1. 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
  2. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  3. 2020When Do Curricula Work?cited by 106
  4. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
  5. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  6. 2017Covariances, Robustness, and Variational Bayescited by 40
  7. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  8. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  9. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  10. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  11. 2025You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisationcited by 17
  12. 2024Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learningcited by 16
  13. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
  14. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
  15. 2024Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networkscited by 9
  16. 2025The Loss Kernel: A Geometric Probe for Deep Learning Interpretabilitycited by 9
  17. 2025Training Data Attribution (TDA): Examining Its Adoption & Use Casescited by 5
  18. 2025Structure Development in List-Sorting Transformerscited by 4
  19. 2025Emergence of Computational Structure in a Neural Network Physics Simulatorcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  2. 2026Dead Directions: Geometric Singular Learningcited by 1
  3. 2026Do Neural Networks Preserve Case Structure? Case-Based Decomposition, Interpretation, and Decision Consistencycited by 1
  4. 2026From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attributioncited by 0
  5. 2026The Free-Recipe Limit: Every Recipe Effect Measures Which Premise of an Idealised Learner Brokecited by 0
  6. 2026What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Usecited by 0
  7. 2026Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic scorecited by 0