mlsanityData to 7 Oct 2026

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models

arXiv 2604.16809 ↗First posted 18 Apr 2026stat.ML cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We study one stylized hypothesis: normalization can postpone instability by gradually increasing the effective learning rate during otherwise stable descent. To test this hypothesis at theorem level, we analyze batch-normalized linear models. Our flagship result concerns whitened square-loss linear regression, where we derive explicit no-rising-edge and delayed-onset conditions, bound the waiting time to directional onset, and show that the rising edge self-stabilizes within finitely many iterations. Combined with a square-loss decomposition, this yields a concrete delayed-spike mechanism in the whitened regime. For logistic regression, under highly restrictive active-margin assumptions, we prove only a supporting finite-horizon directional precursor in a knife-edge regime, with an optional appendix-only loss lower bound under an extra non-degeneracy condition. The paper should therefore be read as a stylized mechanism study rather than a general explanation of neural-network loss spikes. Within that scope, the results isolate one concrete delayed-instability pathway induced by batch normalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

31 of 43 references.

References · 31

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Layer Normalizationcited by 5,579
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  8. 2022Understanding the unstable convergence of gradient descentcited by 84
  9. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  10. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  11. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  12. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
19 more
  1. 2019A Mean Field Theory of Batch Normalizationcited by 58
  2. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  3. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  4. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  5. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  6. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  7. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  8. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  9. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  10. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  11. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  12. 2024On the Nonlinearity of Layer Normalizationcited by 9
  13. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  14. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  15. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  16. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  17. 2025ZClip: Adaptive Spike Mitigation for LLM Pre-Trainingcited by 8
  18. 2020Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Netscited by 5
  19. 2025AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clippingcited by 3

Built on by

Not cited yet.