mlsanityData to 14 Oct 2026

TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control

arXiv 2510.27527 ↗First posted 31 Oct 2025cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) training is prohibitively expensive, driving interest in low-precision fully-quantized training (FQT). While novel 4-bit formats like NVFP4 offer substantial efficiency gains, achieving near-lossless training at such low precision remains challenging. We introduce TetraJet-v2, an end-to-end 4-bit FQT method that leverages NVFP4 for activations, weights, and gradients in all linear layers. We identify two critical issues hindering low-precision LLM training: weight oscillation and outliers. To address these, we propose: 1) an unbiased double-block quantization method for NVFP4 linear layers with practically optimal convergence in LLM training, 2) OsciReset, the first effective algorithm to suppress LLMs' weight oscillation bottleneck, and 3) OutControl, a mix-precision algorithm to retain outlier accuracy. TetraJet-v2 outperforms prior methods on FP4 pre-training for LLMs across models up to 370M parameters trained up to 212B tokens, reducing the performance gap to BF16 by an average of 51.3% while enabling an 1.67x end-to-end speedup over FP8. The code is available at https://github.com/thu-ml/TetraJet-v2-NVFP4Training.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
29 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2019Root Mean Square Layer Normalizationcited by 1,480
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  10. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  11. 2024Massive Activations in Large Language Modelscited by 294
  12. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  13. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  14. 2023Microscaling Data Formats for Deep Learningcited by 152
  15. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  16. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  17. 2025Pretraining Large Language Models with NVFP4cited by 76
  18. 2025Artificial Intelligence Index Report 2025cited by 64
  19. 2023Training Transformers with 4-bit Integerscited by 60
  20. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  21. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  22. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  23. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  24. 2025Training LLMs with MXFP4cited by 37
  25. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  26. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  27. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 21
  28. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 18
  29. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  2. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 21
  3. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 8
  4. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  5. 2026FAAR: Format-Aware Adaptive Rounding for NVFP4cited by 4
  6. 2026Adaptive Block-Scaled Data Typescited by 4
  7. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 3
  8. 2026ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initializationcited by 2
  9. 2026Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipecited by 2
  10. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 2
  11. 2026OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimensioncited by 1
  12. 2026Why Does Post-Training Quantization Work?cited by 1
2 more
  1. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0
  2. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
  2. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
  3. 2025FP4 All the Way: Fully Quantized Training of LLMs
  4. 2023SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
  5. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
  6. 2026OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization
  7. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
  8. 2024OutlierTune: Efficient Channel-Wise Quantization for Large Language Models