mlsanityData to 7 Oct 2026

Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization

arXiv 2403.12422 ↗First posted 19 Mar 2024cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Pretraining transformers are generally time-consuming. Fully quantized training (FQT) is a promising approach to speed up pretraining. However, most FQT methods adopt a quantize-compute-dequantize procedure, which often leads to suboptimal speedup and significant performance degradation when used in transformers due to the high memory access overheads and low-precision computations. In this work, we propose Jetfire, an efficient and accurate INT8 training method specific to transformers. Our method features an INT8 data flow to optimize memory access and a per-block quantization method to maintain the accuracy of pretrained transformers. Extensive experiments demonstrate that our INT8 FQT method achieves comparable accuracy to the FP16 training baseline and outperforms the existing INT8 training works for transformers. Moreover, for a standard transformer block, our method offers an end-to-end training speedup of 1.42x and a 1.49x memory reduction compared to the FP16 baseline. Our code is open sourced at https://github.com/thu-ml/Jetfire-INT8Training.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

34 of 48 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2016Layer Normalizationcited by 5,579
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2017Mixed Precision Trainingcited by 1,135
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
22 more
  1. 2019Learned Step Size Quantizationcited by 604
  2. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  3. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  4. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  7. 2021Post-Training Quantization for Vision Transformercited by 274
  8. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  10. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  11. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  12. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  13. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  14. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 77
  15. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 77
  16. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  17. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  18. 2023Training Transformers with 4-bit Integerscited by 58
  19. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  20. 2021Distribution Adaptive INT8 Quantization for Training CNNscited by 27
  21. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  22. 2022MKQ-BERT: Quantized BERT with 4-bits Weights and Activationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 7.

30 papers.

Sort:

By area:

  1. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  2. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  3. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  4. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  5. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  6. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
  7. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  8. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  9. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
  10. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  11. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  12. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
18 more
  1. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  2. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  3. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Modelscited by 10
  4. 2025SAGE: A Framework of Precise Retrieval for RAGcited by 8
  5. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  6. 2025Beyond Outliers: A Study of Optimizers Under Quantizationcited by 6
  7. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
  8. 2026STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Trainingcited by 4
  9. 20241-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bitcited by 3
  10. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 3
  11. 2025pychop: Emulating Low-Precision Arithmetic in Numerical Methods and Neural Networkscited by 1
  12. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  13. 2025NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devicescited by 0
  14. 2025MoR: Mixture Of Representations For Mixed-Precision Trainingcited by 0
  15. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Trainingcited by 0
  16. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0
  17. 2026AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMscited by 0
  18. 2026A Target-Centric Survey of Quantization-Aware Trainingcited by 0