mlsanityData to 14 Oct 2026

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

arXiv 2510.25602 ↗First posted 29 Oct 2025cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats to handle the pervasive activation outliers in Large Language Models (LLMs). Despite this industry trend, a unified comparison of FP and integer (INT) quantization across varying granularities has been missing, leaving algorithm and hardware co-design without clear guidance. This paper fills that gap by systematically investigating the trade-offs between FP and INT formats. We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced. Our comprehensive comparison demonstrates that for popular 8-bit fine-grained formats (e.g., MX with block size 32), MXINT8 is superior to its FP counterpart in both algorithmic accuracy and hardware efficiency. However, for 4-bit formats, FP (e.g., MXFP4, NVFP4) often holds an accuracy advantage , though we show that NVINT4 can surpass NVFP4 when outlier-mitigation techniques like Hadamard rotation are applied. We also introduce a symmetric clipping method that resolves gradient bias in fine-grained low-bit INT training, enabling nearly lossless performance for MXINT8 training. These findings challenge the current hardware trajectory, demonstrating that a one-size-fits-all FP approach is suboptimal and advocating that fine-grained INT formats, particularly MXINT8, offer a better balance of accuracy, power, and efficiency for future AI accelerators.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

31 of 42 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
19 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  4. 20242 OLMo 2 Furiouscited by 340
  5. 2024Massive Activations in Large Language Modelscited by 294
  6. 2023Microscaling Data Formats for Deep Learningcited by 152
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  8. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  9. 2024Scaling Laws for Precisioncited by 95
  10. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  11. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  12. 2025Training LLMs with MXFP4cited by 37
  13. 2024Accuracy is Not All You Needcited by 31
  14. 2025Recipes for Pre-training LLMs with MXFP8cited by 22
  15. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 21
  16. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 19
  17. 2025Scaling Law for Quantization-Aware Trainingcited by 17
  18. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 12
  19. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 16.

18 papers.

Sort:

By area:

  1. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 35
  2. 2026Flow caching for autoregressive video generationcited by 25
  3. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 21
  4. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  5. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  6. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  7. 2026Adaptive Block-Scaled Data Typescited by 4
  8. 2026ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initializationcited by 2
  9. 2026Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipecited by 2
  10. 2026Quantizing Recursive Reasoning Modelscited by 2
  11. 2026FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scalingcited by 2
  12. 2026QuantClaw: Precision Where It Matters for OpenClawcited by 1
6 more
  1. 2026GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptationcited by 1
  2. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0
  3. 2026Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decodingcited by 0
  4. 2026Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inferencecited by 0
  5. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Designcited by 0
  6. 2026Disaggregated Quantization: Specializing LLM Prefill and Decodecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  2. 2023INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformers
  3. 2026Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction
  4. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformers
  5. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks
  6. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
  7. 2023Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs
  8. 2025FP4 All the Way: Fully Quantized Training of LLMs