mlsanityData to 14 Oct 2026

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

arXiv 2606.14598 ↗First posted 12 Jun 2026cs.LGcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization, yet on consumer Ampere GPUs it is frequently slower than the FP8 and NF4 alternatives it is meant to beat. We trace this to a software artifact: the production "INT8" forward quantizes weights and activations only to immediately dequantize them back to bf16 and run a bf16 matrix multiply, never engaging the GPU's INT8 tensor cores, so the hardware's compute advantage is left entirely unrealized. We close this gap with a single fused Triton INT8 GEMM (int8xint8->int32 on Ampere tensor cores, with per-token x per-channel dequantization and bias folded into the epilogue, autotuned per GEMM shape) dropped into the Ideogram 4.0 diffusion transformer's linear layers in place of the dequantize-to-bf16 path. In the kernel, the int8xint8->int32 accumulation is bit-exact against torch._int_mm and the dequantized output matches the reference at cosine similarity 1.0 with no NaNs, running 2.8-4.2x faster than bf16 per GEMM. End to end it delivers a ~1.1x (~9-10%) speedup at 768px, and at 1024px it generates an image in 156.5 s on a single RTX 3090, faster than the single-card NF4 (164.5 s) and FP8 (172.9 s) baselines, at no measurable quality cost on these point estimates (PickScore/CLIPScore). INT8 thus goes from the slowest variant to the fastest, and 1024px becomes single-GPU feasible. The primary speed criterion (beat FP8, by ~9.5%) is comfortably met; the NF4 margin (~4.9%, single-run n=4) is within run-to-run variance we did not quantify and is best read as consistent with meeting the stretch target. We close with an honest deployment map: the win is specific to consumer Ampere, and on A100 and B200 the same kernel loses to those cards' fast native bf16/FP8 paths.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 21 of its 24 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

21 of 24 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Holding the FP8 Quality Ceiling at 8-Bit Weights and Activations: INT8 and GGUF Post-Training Quantization of Ideogram 4.0 for Consumer GPUs10 Jun
  2. 2026DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers16 May
  3. 2025DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization17 Jul
  4. 2025MixDiT: Accelerating Image Diffusion Transformer Inference with Mixed-Precision MX Quantization11 Apr
  5. 2025Efficient Diffusion Models: A Survey3 Feb
  6. 2025SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity26 Jan
  7. 2024PQD: Post-training Quantization for Efficient Diffusion Models30 Dec
  8. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models7 Nov

Other references · 13

  1. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  2. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  3. 20241.58-bit FLUXcited by 31
  4. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22
  5. 2026AutoKernel: Autonomous GPU Kernel Optimization via Iterative Agent-Driven Searchcited by 18
  6. 2024QQQ: Quality Quattuor-Bit Quantization for Large Language Modelscited by 16
  7. 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMscited by 12
  8. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Modelscited by 10
  9. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Servingcited by 8
  10. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  11. 2025Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computationcited by 3
  12. 2025FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Accelerationcited by 2
1 more
  1. 2026Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference
  2. 2026Performance Analysis and Optimization of 3D Generative Diffusion Models across GPU Architectures
  3. 2026DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction
  4. 2026Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators
  5. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
  6. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas
  7. 2023Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations
  8. 2025Energy Scaling Laws for Diffusion Models: Quantifying Compute in Image Generation