mlsanityData to 7 Oct 2026

Accurate INT8 Training Through Dynamic Block-Level Fallback

arXiv 2503.08040 ↗First posted 11 Mar 2025cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved remarkable success across various AI applications but face significant training costs. Low-bit training, such as INT8 training, can leverage computational units with higher throughput, and has already demonstrated its effectiveness on GPT2 models with block-level quantization. However, it struggles with modern Transformer variants incorporating GLU units. This is because those variants demonstrate complex distributions of activation outliers. To address the challenge, we propose Fallback Quantization, implementing mixed-precision GEMM that dynamically falls back 8-bit to 16-bit for activation blocks containing outliers. Experiments show that our approach is robustly competent in both fine-tuning and pretraining settings. Moreover, our method achieves a 1.57x end-to-end training speedup on RTX4090 GPUs.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

38 of 49 references.

References · 38

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
26 more
  1. 2017Mixed Precision Trainingcited by 1,135
  2. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  4. 2016Language Modeling with Gated Convolutional Networkscited by 688
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  8. 2015Deep Learning with Limited Numerical Precisioncited by 368
  9. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  10. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  11. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  12. 2022FP8 Formats for Deep Learningcited by 187
  13. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  14. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  15. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  16. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  17. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 77
  18. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  19. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  20. 2024Scaling FP8 training to trillion-token LLMscited by 63
  21. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  22. 2023Training Transformers with 4-bit Integerscited by 58
  23. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  24. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  25. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  26. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  2. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  3. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  4. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  5. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  6. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
  7. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  8. 2026SageBwd: A Trainable Low-bit Attentioncited by 7
  9. 2026HiFloat4 Format for Language Model Pre-training on Ascend NPUscited by 4
  10. 2026Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generationcited by 4
  11. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  12. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
2 more
  1. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0
  2. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0