mlsanityData to 7 Oct 2026

Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

arXiv 2607.04422 ↗First posted 5 Jul 2026cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent NVFP4 pretraining work has primarily optimized Transformer linear projections, leaving persistent optimizer states, optimizer computation, and low-precision attention forward--backward paths less explored. We present Full-Stack FP4, a modular NVFP4 framework with separate recipes for projections, AdamW states, Root/Muon computation, and attention. LoRA-SVD protects a compact projection subspace in BF16 while retaining full-shape NVFP4 computation, reducing the linear-only loss gap from 1.40% to 0.61%. An ordered square-root, tile-mean, and Hadamard pipeline enables stable NVFP4 AdamW momentum storage; shape-dependent coefficients and clipping stabilize direct NVFP4 Root iterations; and mixed-precision attention retains softmax-sensitive operations in BF16. On 3B pretraining with 64B tokens, BF16 and Full-Stack FP4 reach losses of 2.267 and 2.286, a 0.838% gap. Their average zero-shot perplexities are 26.675 and 26.665, respectively, with Full-Stack FP4 averaging 0.10 percentage points lower in accuracy. Native four-block measurements on one RTX 5090 show 2.50--2.83× Root speedups over optimized BF16 and 37.9--42.5% lower AdamW peak memory.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

32 of 39 references.

References · 32

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  5. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  6. 2022FP8 Formats for Deep Learningcited by 187
  7. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  8. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  9. 2023Microscaling Data Formats for Deep Learningcited by 132
  10. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  11. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  12. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
20 more
  1. 2025Pretraining Large Language Models with NVFP4cited by 67
  2. 2024Scaling FP8 training to trillion-token LLMscited by 63
  3. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  4. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  5. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  6. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  7. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  8. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
  9. 2025Training LLMs with MXFP4cited by 36
  10. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  11. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
  12. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  13. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
  14. 2025ROOT: Robust Orthogonalized Optimizer for Neural Network Trainingcited by 13
  15. 2025Effective Quantization of Muon Optimizer Statescited by 10
  16. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  17. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 5
  18. 2026CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programscited by 5
  19. 2025Elucidating the Design Space of FP4 trainingcited by 4
  20. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026UE5M3 FP4 Block Scaling for Stable Language Model Pretrainingcited by 1
  2. 2026Auditing Invisible Weight Updates with Reference Tracescited by 0
  3. 2026KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantizationcited by 0
  4. 2026Pretraining Transformers with Quantized Softmax in Attentioncited by 0
  5. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMscited by 0
  6. 2026Format-Aware Fusion for Fast FP4 Pretrainingcited by 0
  7. 2026Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantizationcited by 0