# HiFloat4 Format for Language Model Inference

arXiv: https://arxiv.org/abs/2602.11287 · HTML: https://mlsanity.com/paper/2602.11287/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [AI and Memory Wall](https://mlsanity.com/paper/2403.14123/) (2024)
- [Pretraining Large Language Models with NVFP4](https://mlsanity.com/paper/2509.25149/) (2025)
- [LongCat-Flash Technical Report](https://mlsanity.com/paper/2509.01322/) (2025)
- [With Shared Microexponents, A Little Shifting Goes a Long Way](https://mlsanity.com/paper/2302.08007/) (2023)
- [Ascend HiFloat8 Format for Deep Learning](https://mlsanity.com/paper/2409.16626/) (2024)

## Built on by (most cited first, 11 of 11)

- [HiFloat4 Format for Language Model Pre-training on Ascend NPUs](https://mlsanity.com/paper/2604.08826/) (2026)
- [VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation](https://mlsanity.com/paper/2604.12798/) (2026)
- [Collaborative Few-Step Distillation and Low-Bit Quantization for Wan2.2 Dual-Expert Video Diffusion Models](https://mlsanity.com/paper/2606.00658/) (2026)
- [OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension](https://mlsanity.com/paper/2604.12782/) (2026)
- [HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference](https://mlsanity.com/paper/2607.04302/) (2026)
- [Tail-Aware HiFloat4: W4A4 Post-Training Quantization for Wan2.2](https://mlsanity.com/paper/2605.26628/) (2026)
- [SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference](https://mlsanity.com/paper/2606.26587/) (2026)
- [HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models](https://mlsanity.com/paper/2607.26515/) (2026)
- [Transforms for LLM Quantization: The Great Inversion and Format Co-Design](https://mlsanity.com/paper/2608.25188/) (2026)
- [Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs](https://mlsanity.com/paper/2608.26581/) (2026)
- [DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving](https://mlsanity.com/paper/2610.13328/) (2026)

## Abstract (arXiv)

This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning. Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value. The metadata specifies a three-level scaling hierarchy, capturing inter- and intra-group dynamic range while improving the utilization of the representational space. In addition, the large 64-element group size enables matrix multiplications to be executed in a highly fixed-point manner, significantly reducing hardware area and power consumption. To evaluate the proposed format, we conducted inference experiments on several language models, including LLaMA, Qwen, Mistral, DeepSeek-V3.1 and LongCat. Results show that HiF4 achieves higher average accuracy than the state-of-the-art NVFP4 format across multiple models and diverse downstream tasks.