mlsanityData to 13 Oct 2026

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv 2606.20381 ↗First posted 18 Jun 2026cs.AIcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 10 of its 27 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

10 of 27 references.

References · 10

  1. 2022FP8 Formats for Deep Learningcited by 192
  2. 2023Microscaling Data Formats for Deep Learningcited by 146
  3. 2025Pretraining Large Language Models with NVFP4cited by 71
  4. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 33
  5. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 20
  6. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 18
  7. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  8. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  9. 2026FAAR: Format-Aware Adaptive Rounding for NVFP4cited by 4
  10. 2026MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representationscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0
  2. 2026LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
  2. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
  3. 2025FP4 All the Way: Fully Quantized Training of LLMs
  4. 2025Elucidating the Design Space of FP4 training
  5. 2026ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization
  6. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
  7. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
  8. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference