# Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv: https://arxiv.org/abs/2606.20381 · HTML: https://mlsanity.com/paper/2606.20381/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 10 of 10)

- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [Pretraining Large Language Models with NVFP4](https://mlsanity.com/paper/2509.25149/) (2025)
- [Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling](https://mlsanity.com/paper/2512.02010/) (2025)
- [Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation](https://mlsanity.com/paper/2601.22813/) (2026)
- [INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats](https://mlsanity.com/paper/2510.25602/) (2025)
- [TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control](https://mlsanity.com/paper/2510.27527/) (2025)
- [HiFloat4 Format for Language Model Pre-training on Ascend NPUs](https://mlsanity.com/paper/2604.08826/) (2026)
- [FAAR: Format-Aware Adaptive Rounding for NVFP4](https://mlsanity.com/paper/2603.22370/) (2026)
- [MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations](https://mlsanity.com/paper/2605.31035/) (2026)

## Built on by (most cited first, 2 of 2)

- [Format-Aware Fusion for Fast FP4 Pretraining](https://mlsanity.com/paper/2610.00053/) (2026)
- [LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization](https://mlsanity.com/paper/2610.03959/) (2026)

## Abstract (arXiv)

FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.