mlsanityData to 7 Oct 2026

INT-FlashAttention: Enabling Flash Attention for INT8 Quantization

arXiv 2409.16997 ↗First posted 25 Sep 2024cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

As the foundation of large language models (LLMs), self-attention module faces the challenge of quadratic time and memory complexity with respect to sequence length. FlashAttention accelerates attention computation and reduces its memory usage by leveraging the GPU memory hierarchy. A promising research direction is to integrate FlashAttention with quantization methods. This paper introduces INT-FlashAttention, the first INT8 quantization architecture compatible with the forward workflow of FlashAttention, which significantly improves the inference speed of FlashAttention on Ampere GPUs. We implement our INT-FlashAttention prototype with fully INT8 activations and general matrix-multiplication (GEMM) kernels, making it the first attention operator with fully INT8 input. As a general token-level post-training quantization framework, INT-FlashAttention is also compatible with other data formats like INT4, etc. Experimental results show INT-FlashAttention achieves 72% faster inference speed and 82% smaller quantization error compared to standard FlashAttention with FP16 and FP8 data format.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

21 of 26 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  8. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  9. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  12. 2022FP8 Formats for Deep Learningcited by 187
9 more
  1. 2018Online normalizer calculation for softmaxcited by 154
  2. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  3. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  4. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  5. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  6. 2024Evaluating Quantized Large Language Modelscited by 75
  7. 2024Fast On-device LLM Inference with NPUscited by 58
  8. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  9. 2024TernaryLLM: Ternarized Large Language Modelcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  2. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  3. 2025AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Upcited by 6
  4. 2026VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentioncited by 1
  5. 2025Arithmetic-Intensity-Aware Quantizationcited by 0
  6. 2026On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passescited by 0
  7. 2026QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attentioncited by 0
  8. 2026TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Accelerationcited by 0
  9. 2026Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Accelerationcited by 0