mlsanityData to 14 Oct 2026

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

arXiv 2604.25306 ↗First posted 28 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

FlashAttention improves efficiency through tiling, but its online softmax still relies on floating-point arithmetic for numerical stability, making full quantization difficult. We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison. To address these challenges, we propose textitQFlash, an end-to-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel. On seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73× speedup over I-ViT and up to 8.69× speedup on Swin, while reducing energy consumption by 18.8% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under per-tensor quantization. Our code is publicly available at https://github.com/EfficientCompLab/qflash.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

16 of 26 references.

References · 16

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  4. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  5. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 179
  6. 2018Online normalizer calculation for softmaxcited by 162
  7. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  8. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 137
  9. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  10. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 30
  11. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 23
  12. 2024CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTscited by 17
4 more
  1. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 11
  2. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  3. 2024Mixed Non-linear Quantization for Vision Transformerscited by 3
  4. 2019Low Overhead Instruction Latency Characterization for NVIDIA GPGPUscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
  2. 2025IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference
  3. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration
  4. 2025Low-Bit Integerization of Vision Transformers using Operand Reordering for Efficient Hardware
  5. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer
  6. 2026Dispatch-Aware Ragged Attention for Pruned Vision Transformers
  7. 2026From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
  8. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer