mlsanityData to 13 Oct 2026

Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability

arXiv 2606.28116 ↗First posted 26 Jun 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal. We study mechanism-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures. For low-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first-order term becomes abnormal before the loss fully collapses. For MoE routers, we derive indicators from their role in expert selection. Our fault-injection experiments on low-precision attention, large learning-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

25 of 33 references.

References · 25

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 737
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 693
  7. 2018On Lazy Training in Differentiable Programmingcited by 675
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 456
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 419
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 413
  11. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  12. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 200
13 more
  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  2. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 157
  3. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  4. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 82
  5. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 81
  6. 2026Attention Residualscited by 58
  7. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  8. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  9. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 30
  10. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  11. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12
  12. 2025Why Low-Precision Transformer Training Fails: An Analysis on Flash Attentioncited by 12
  13. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026One QK Channel, Many Sources: Tracing Low-Precision Attention Collapsecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
  2. 2026MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training
  3. 2026"I May Not Have Articulated Myself Clearly": Diagnosing Dynamic Instability in LLM Reasoning at Inference Time
  4. 2026Fast and Accurate Probing of In-Training LLMs' Downstream Performances
  5. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
  6. 2025Language Models Can Predict Their Own Behavior
  7. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
  8. 2026FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness