mlsanityData to 14 Oct 2026

Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization

arXiv 2509.23202 ↗First posted 27 Sep 2025cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

The recent hardware-accelerated microscaling 4-bit floating-point formats such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to revolutionize large language model (LLM) inference. Yet, their practical benefits remain unproven. We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization, revealing gaps between their promise and real-world performance. Our analysis shows that state-of-the-art methods struggle with FP4, due to two key issues: (1) NVFP4's small group size provably neutralizes traditional outlier mitigation techniques; (2) MXFP4's power-of-two scale quantization severely degrades accuracy due to high induced error. To bridge this gap, we introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm that tailors the quantization process to FP4's unique properties, by using block-wise Hadamard transforms and format-specific optimizations. We support our proposal with a set of high-performance GPU kernels that enable the MR-GPTQ format with negligible overhead, by rotation fusion into the weights, and fast online computation of the activations. This leads to speedups vs. FP16 of up to 3.6x layer-wise, and 2.2x end-to-end on NVIDIA B200, and of 6x layer-wise and 4x end-to-end on RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches or outperforms state-of-the-art accuracy, significantly boosting MXFP4, to the point where it can near the accuracy that of NVFP4. We conclude that, while FP4 is not an automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock a new frontier of accuracy-performance trade-offs.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

36 of 56 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
24 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  2. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  4. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  6. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  7. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  10. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  13. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  14. 2024Evaluating Quantized Large Language Modelscited by 77
  15. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  16. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  17. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  18. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  19. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  20. 2024What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbationcited by 23
  21. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  22. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  23. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 8
  24. 2025Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Freecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 29.

32 papers.

Sort:

By area:

  1. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  2. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 17
  3. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 15
  4. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  5. 2026Pushing the Limits of Block Rotations in Post-Training Quantizationcited by 11
  6. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 10
  7. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  8. 2026DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantizationcited by 8
  9. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  10. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  11. 2026FAAR: Format-Aware Adaptive Rounding for NVFP4cited by 4
  12. 2026Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMscited by 4
20 more
  1. 2026Optimal Post-Training Quantization Scales and Where to Find Themcited by 3
  2. 2026LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMscited by 2
  3. 2026Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4cited by 2
  4. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 2
  5. 2026Quantizing Recursive Reasoning Modelscited by 2
  6. 2026FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scalingcited by 2
  7. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  8. 2026Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimatecited by 1
  9. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  10. 2026Timestep-Aware SVDQuant-GPTQ for W4A4 Quantization of Wan2.2-I2Vcited by 1
  11. 2026GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptationcited by 1
  12. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0
  13. 2026Not All NVFP4 QAT Recipes Are Equal: How Architecture and Scale Shape Model Quality for Anomaly Segmentationcited by 0
  14. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inferencecited by 0
  15. 2026PRQuant: Permutation Residual Quantization for Low-Overhead Inferencecited by 0
  16. 2026Disaggregated Quantization: Specializing LLM Prefill and Decodecited by 0
  17. 2026LLMs are General Asynchronous Agentscited by 0
  18. 2026Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inferencecited by 0
  19. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Expertscited by 0
  20. 2026CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Cachescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction
  2. 2025Block Rotation is All You Need for MXFP4 Quantization
  3. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference
  4. 2026OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization
  5. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
  6. 2025FP4 All the Way: Fully Quantized Training of LLMs
  7. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  8. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs