# Is Finer Better? The Limits of Microscaling Formats in Large Language Models

arXiv: https://arxiv.org/abs/2601.19026 · HTML: https://mlsanity.com/paper/2601.19026/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 21 of 21)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training](https://mlsanity.com/paper/2505.11594/) (2025)
- [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021)
- [Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning](https://mlsanity.com/paper/2501.03035/) (2025)
- [Towards Fully 8-bit Integer Inference for the Transformer Model](https://mlsanity.com/paper/2009.08034/) (2020)
- [AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference](https://mlsanity.com/paper/2411.09909/) (2024)
- [BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration](https://mlsanity.com/paper/2411.11745/) (2024)
- [BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference](https://mlsanity.com/paper/2501.01144/) (2025)
- [Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Models](https://mlsanity.com/paper/2412.19821/) (2024)

## Built on by (most cited first, 8 of 8)

- [Quantizing Recursive Reasoning Models](https://mlsanity.com/paper/2607.16237/) (2026)
- [UE5M3 FP4 Block Scaling for Stable Language Model Pretraining](https://mlsanity.com/paper/2609.02846/) (2026)
- [Finer is Better (with the Right Scaling)](https://mlsanity.com/paper/2605.08565/) (2026)
- [A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models](https://mlsanity.com/paper/2605.14929/) (2026)
- [Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats](https://mlsanity.com/paper/2606.09686/) (2026)
- [HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference](https://mlsanity.com/paper/2609.00450/) (2026)
- [Format-Aware Fusion for Fast FP4 Pretraining](https://mlsanity.com/paper/2610.00053/) (2026)
- [VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Models](https://mlsanity.com/paper/2610.03218/) (2026)

## Abstract (arXiv)

Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.