mlsanityData to 14 Oct 2026

Is Finer Better? The Limits of Microscaling Formats in Large Language Models

arXiv 2601.19026 ↗First posted 26 Jan 2026cs.LG cs.AR cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

21 of 26 references.

References · 21

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2015Deep Learning with Limited Numerical Precisioncited by 376
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  10. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  11. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  12. 2023Microscaling Data Formats for Deep Learningcited by 152
9 more
  1. 2022FP8 Quantization: The Power of the Exponentcited by 88
  2. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  3. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  4. 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoningcited by 37
  5. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  6. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 19
  7. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 18
  8. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 11
  9. 2024Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Quantizing Recursive Reasoning Modelscited by 2
  2. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 2
  3. 2026Finer is Better (with the Right Scaling)cited by 0
  4. 2026A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Modelscited by 0
  5. 2026Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formatscited by 0
  6. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inferencecited by 0
  7. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0
  8. 2026VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formats
  2. 2024Scaling Laws For Mixed Quantization
  3. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
  4. 2026Stable FP4 Training via Transposition-Invariant Block Quantization
  5. 2024Post Training Quantization of Large Language Models with Microscaling Formats
  6. 2026Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
  7. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
  8. 2025Compression Scaling Laws:Unifying Sparsity and Quantization