Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4
Not on the map. Why?
Abstract (arXiv)
Quantization addresses the high resource demand for large language models (LLMs) by alleviating memory pressure and bandwidth congestion and providing significantly scaled compute power with a tolerable impact on accuracy. Four-bit floating point (FP4), the lowest-precision format that preserves essential numerical properties such as exponent and sign, has begun to be adopted in cutting-edge architectures, including Blackwell and AMD CDNA, to support LLM quantization and reduce deployment costs. Although aggressive quantization can yield efficiency gains, the quantization sensitivity of within-transformer layers and whether these sensitivities generalize across existing FP4 formats and model scales remain underexplored. To elucidate quantization sensitivity, this study conducts a systematic analysis of two FP4 formats, MXFP4 and NVFP4, across three Qwen2.5 model scales (0.5B, 7B, and 14B), using controlled component-wise and block-wise isolation methodologies. We observe that MLP up- and down-projection layers consistently dominate in terms of sensitivity, while gate and attention projections are moderately and substantially less sensitive to FP4 quantization, respectively. We further find that sensitivity does not universally localize to the final blocks, but early blocks can be highly sensitive, particularly under MXFP4. Our results provide a diagnostic characterization of the inference behavior of FP4 across components, depths, and FP4 formats.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 5 Mar 2026.
Builds on
24 of 31 references.
References · 24
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2020GLU Variants Improve Transformercited by 1,724
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2017Mixed Precision Trainingcited by 1,135
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
12 more
- 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
- 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
- 2025Pretraining Large Language Models with NVFP4cited by 67
- 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
- 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
- 2025Training LLMs with MXFP4cited by 36
- 2024Investigating Layer Importance in Large Language Modelscited by 35
- 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 30
- 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 14
- 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
- 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
- 2025Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Trainingcited by 9
Built on by
Citing papers per year: 2026: 2.
2 papers.