mlsanityData to 14 Oct 2026

Oscillation-Reduced MXFP4 Training for Vision Transformers

arXiv 2502.20853 ↗First posted 28 Feb 2025cs.LG cs.AI cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Pre-training Transformers in FP4 precision is becoming a promising approach to gain substantial speedup, but it comes with a considerable loss of accuracy. Microscaling (MX) data format provides a fine-grained per-group quantization method to improve the representation ability of the FP4 format and is supported by the next-generation Blackwell GPU architecture. However, training with MXFP4 data format still results in significant degradation and there is a lack of systematic research on the reason. In this work, we propose a novel training method TetraJet for a more accurate FP4 training. We comprehensively evaluate all of the quantizers involved in the training, and identify the weight oscillation problem in the forward pass as the main source of the degradation in MXFP4 training. Therefore, we introduce two novel methods, EMA Quantizer (Q-EMA) and Adaptive Ramping Optimizer (Q-Ramping), to resolve the oscillation problem. Extensive experiments on Vision Transformers demonstrate that TetraJet consistently outperforms the existing 4-bit training methods, and Q-EMA & Q-Ramping can provide additional enhancement by effectively reducing oscillation. We decreased the accuracy degradation by more than 50% compared to the baseline, and can even achieve competitive performance compared to full precision training. The codes are available at https://github.com/thu-ml/TetraJet-MXFP4Training

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

19 of 26 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2017Mixed Precision Trainingcited by 1,156
  7. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  8. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  9. 2022FP8 Formats for Deep Learningcited by 196
  10. 2023Microscaling Data Formats for Deep Learningcited by 152
  11. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  12. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 79
7 more
  1. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  2. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  3. 2023Training Transformers with 4-bit Integerscited by 60
  4. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  5. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  6. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  7. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025Pretraining Large Language Models with NVFP4cited by 76
  2. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 18
  3. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  4. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  5. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
  6. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 10
  7. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
  8. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 5
  9. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  10. 2025Elucidating the Design Space of FP4 trainingcited by 4
  11. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  12. 2026Quantizing Recursive Reasoning Modelscited by 2
9 more
  1. 2026MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attentioncited by 2
  2. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 2
  3. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  4. 2025Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Trainingcited by 0
  5. 2025Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integrationcited by 0
  6. 2026MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocationcited by 0
  7. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inferencecited by 0
  8. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0
  9. 2026VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
  2. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization
  3. 2026Pretraining large language models with MXFP4 on Native FP4 Hardware
  4. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
  5. 2025Recipes for Pre-training LLMs with MXFP8
  6. 2026ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization
  7. 2025Block Rotation is All You Need for MXFP4 Quantization
  8. 2026Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction