Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
Not on the map. Why?
Abstract (arXiv)
Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining, focusing on where they localize, why they occur, and how they evolve temporally. We find that, compared with Softmax Attention (SA), Linear Attention (LA) reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization. Our analysis attributes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with "post-QK" operations exhibiting higher sensitivity to quantization. Notably, outliers evolve from transient spikes early in training to a small set of persistent hot channels (i.e., channels with persistently large magnitudes) in later stages. Based on these findings, we introduce Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels and reinjects residuals using hardware-efficient kernels. We then develop CHON, an NVFP4 training recipe integrating HCP with post-QK operation protection. On GLA-1.3B model trained for 60B tokens, CHON reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 51 of its 64 references that are in the map. First posted on arXiv 2 Feb 2026.
Builds on
51 of 64 references.
References · 51
- 2017Attention Is All You Needcited by 59,932
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
- 2020GLU Variants Improve Transformercited by 1,724
- 2020Rethinking Attention with Performerscited by 1,584
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2017Crowdsourcing Multiple Choice Science Questionscited by 742
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
- 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
- 2021Random Feature Attentioncited by 350
- 2024Massive Activations in Large Language Modelscited by 277
- 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
- 2022cosFormer: Rethinking Softmax in Attentioncited by 212
- 2023Microscaling Data Formats for Deep Learningcited by 132
- 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
- 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
- 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
- 2022The Devil in Linear Transformercited by 98
- 2023FP8-LM: Training FP8 Large Language Modelscited by 75
- 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
- 2025Pretraining Large Language Models with NVFP4cited by 67
- 2024Scaling FP8 training to trillion-token LLMscited by 63
- 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
- 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
- 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
- 2025Training LLMs with MXFP4cited by 36
- 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
- 2025Systematic Outliers in Large Language Modelscited by 32
- 2025Superposition Yields Robust Neural Scalingcited by 30
- 2022Vicinity Vision Transformercited by 22
- 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
- 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
- 2023Linear Log-Normal Attention with Unbiased Concentrationcited by 11
- 2025Metis: Training LLMs with FP4 Quantizationcited by 9
- 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.