mlsanityData to 7 Oct 2026

Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps

arXiv 2601.21624 ↗First posted 29 Jan 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Modern deep-learning training is not memoryless. Updates depend on optimizer moments and averaging, data-order policies (random reshuffling vs with-replacement, staged augmentations and replay), the nonconvex path, and auxiliary state (teacher EMA/SWA, contrastive queues, BatchNorm statistics). This survey organizes mechanisms by source, lifetime, and visibility. It introduces seed-paired, function-space causal estimands; portable perturbation primitives (carry/reset of momentum/Adam/EMA/BN, order-window swaps, queue/teacher tweaks); and a reporting checklist with audit artifacts (order hashes, buffer/BN checksums, RNG contracts). The conclusion is a protocol for portable, causal, uncertainty-aware measurement that attributes how much training history matters across models, data, and regimes.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 84 of its 117 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

84 of 117 references.

References · 84

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Layer Normalizationcited by 5,579
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2018Datasheets for Datasetscited by 1,371
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  11. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  12. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  13. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  14. 2015Prioritized Experience Replaycited by 1,070
  15. 2018Model Cards for Model Reportingcited by 1,068
  16. 2020Adaptive Federated Optimizationcited by 1,036
  17. 2019Deep Leakage from Gradientscited by 1,004
  18. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  19. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  20. 2017Deep Reinforcement Learning that Matterscited by 727
  21. 2018On Lazy Training in Differentiable Programmingcited by 671
  22. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  23. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  24. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  25. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  26. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  27. 2016Learning to learn by gradient descent by gradient descentcited by 616
  28. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  29. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  30. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  31. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  32. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  33. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  34. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  35. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  36. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  37. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  38. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Fiber Criterion for Representation Identifiability in Supervised Learningcited by 2