mlsanityData to 14 Oct 2026

Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

arXiv 2608.19762 ↗First posted 20 Aug 2026cs.LG cs.AI math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

22 of 31 references.

References · 22

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2019On the Convergence of Adam and Beyondcited by 841
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  7. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  8. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  9. 2021Scaling Up Influence Functionscited by 132
  10. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  11. 2019Data Cleansing for Models Trained with SGDcited by 67
  12. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
10 more
  1. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  2. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  3. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  4. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  5. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  6. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  7. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  8. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  9. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
  10. 2026How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelinescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observabilitycited by 0