mlsanityData to 7 Oct 2026

GNMR: Runtime Stability Control for Low-Precision Large Language Model Training

arXiv 2606.00539 ↗First posted 30 May 2026cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators. We formulate this as runtime stability control and present Gradient Norm-to-Mean Ratio (GNMR), a lightweight controller that compares each recoverable unit's current gradient norm with its historical mean. Together with Delta-GNMR for abrupt short-window increases, GNMR maps local risk signals to bounded recovery actions under a hard maxO budget and a short lock interval, without changing the numerical format, kernel, or backend recipe. Across activation-quantization stress, DeepSeek-style recipe-level training, and LLaMA-2 13B fine-tuning, GNMR preserves high-fidelity quality with sparse, budgeted recovery. These results support GNMR as a backend-agnostic controller to improve low-precision training stability while preserving low-cost execution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 52 of its 65 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

52 of 65 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2020On Layer Normalization in the Transformer Architecturecited by 906
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  13. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  14. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  15. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  16. 2018An Empirical Model of Large-Batch Trainingcited by 268
  17. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  18. 2022FP8 Formats for Deep Learningcited by 187
  19. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  20. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  21. 2025Pretraining Large Language Models with NVFP4cited by 67
  22. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  23. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  24. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  25. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  26. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  27. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  28. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  29. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  30. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 13
  31. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 10
  32. 2024To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stabilitycited by 9
  33. 2021Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNscited by 8
  34. 2024EfQAT: An Efficient Framework for Quantization-Aware Trainingcited by 7
  35. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  36. 2025PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimizationcited by 5
  37. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  38. 2023Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)cited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.