mlsanityData to 7 Oct 2026

HiFloat4 Format for Language Model Pre-training on Ascend NPUs

arXiv 2604.08826 ↗First posted 9 Apr 2026cs.LG cs.AI cs.CLcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Training large foundation models at low numerical precision is one of the most promising directions for reducing the compute and memory cost of modern AI. Recent 4-bit floating-point formats such as MXFP4 and NVFP4 can be applied to linear GEMM operations in LLMs, but their limited dynamic range introduces numerical instability that prior work addresses by stacking stabilization mechanisms, typically executed at higher precision and partially eroding the efficiency gains that motivate FP4. In this work, we argue that numerical format design is itself a first-class lever for stable FP4 training, and present the first systematic study of FP4 LLM pretraining on energy-efficient Huawei Ascend NPUs. We compare the recently proposed HiFloat4 (HiF4) format against both MXFP4 and NVFP4, holding one recipe fixed across all three formats across dense (OpenPangu-1B, Llama3-8B) and Mixture-of-Experts (Qwen3-MoE-30B) architectures and executing all linear and expert GEMMs in FP4. At matched storage --- NVFP4 and HiF4 both spend 4.5 bits per value --- the three formats differ far more in what they require before they will train at all than in final accuracy. HiF4 reaches a relative loss of 1.55% with no stabilization, below fully stabilized MXFP4 (1.79%) and below NVFP4 carrying the per-tensor scaling it cannot train without (2.00%); NVFP4 diverges under every combination of stochastic rounding and Hadamard transform we tried. Our results suggest that stable, accurate FP4 training does not require an ever-growing stack of stabilization techniques; it requires the right numerical format.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 33 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

26 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core8 Mar
  2. 2026Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs3 Mar

Other references · 24

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  8. 2022FP8 Formats for Deep Learningcited by 187
  9. 2023Microscaling Data Formats for Deep Learningcited by 132
  10. 2025Pretraining Large Language Models with NVFP4cited by 67
  11. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  12. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
12 more
  1. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
  2. 2025Training LLMs with MXFP4cited by 36
  3. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
  4. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  5. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  6. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  7. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 16
  8. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  9. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  10. 2026HiFloat4 Format for Language Model Inferencecited by 9
  11. 2026M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantizationcited by 7
  12. 2025Elucidating the Design Space of FP4 trainingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipecited by 1
  2. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 1
  3. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0
  4. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats
  2. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats
  3. 2023FP8-LM: Training FP8 Large Language Models
  4. 2025MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving
  5. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
  6. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
  7. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization
  8. 2026Normalized Architectures are Natively 4-Bit