# HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization

arXiv: https://arxiv.org/abs/2405.19751 · HTML: https://mlsanity.com/paper/2405.19751/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 25 of 25)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2309.14717/) (2023)
- [FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2111.13824/) (2021)
- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer](https://mlsanity.com/paper/2301.11798/) (2023)
- [DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation](https://mlsanity.com/paper/2307.01831/) (2023)
- [EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models](https://mlsanity.com/paper/2310.03270/) (2023)
- [LLM-FP4: 4-Bit Floating-Point Quantized Transformers](https://mlsanity.com/paper/2310.16836/) (2023)
- [FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding](https://mlsanity.com/paper/2110.15456/) (2021)
- [AFPQ: Asymmetric Floating Point Quantization for LLMs](https://mlsanity.com/paper/2311.01792/) (2023)
- [Term Revealing: Furthering Quantization at Run Time on Quantized DNNs](https://mlsanity.com/paper/2007.06389/) (2020)

## Built on by (most cited first, 17 of 17)

- [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://mlsanity.com/paper/2411.05007/) (2024)
- [1.58-bit FLUX](https://mlsanity.com/paper/2412.18653/) (2024)
- [DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing](https://mlsanity.com/paper/2409.07756/) (2024)
- [Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers](https://mlsanity.com/paper/2510.11538/) (2025)
- [FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformers](https://mlsanity.com/paper/2503.15465/) (2025)
- [Diffusion Model Quantization: A Review](https://mlsanity.com/paper/2505.05215/) (2025)
- [Q-DiT4SR: Exploration of Detail-Preserving Diffusion Transformer Quantization for Real-World Image Super-Resolution](https://mlsanity.com/paper/2602.01273/) (2026)
- [Faster and Better Alignment for Flow Matching Models via Step-aware Advantages](https://mlsanity.com/paper/2602.01591/) (2026)
- [Q-Drift: Quantization-Aware Drift Correction for Diffusion Model Sampling](https://mlsanity.com/paper/2603.18095/) (2026)
- [ProCache: Constraint-Aware Feature Caching with Selective Computation for Diffusion Transformer Acceleration](https://mlsanity.com/paper/2512.17298/) (2025)
- [Steering Video Diffusion Transformers with Massive Activations](https://mlsanity.com/paper/2603.17825/) (2026)
- [VersaQ-3D: Architecture Support for Visual Geometry Grounded Transformers via Versatile Quantization](https://mlsanity.com/paper/2601.20317/) (2026)
- [Q&C: When Quantization Meets Cache in Efficient Image Generation](https://mlsanity.com/paper/2503.02508/) (2025)
- [ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality](https://mlsanity.com/paper/2510.24787/) (2025)
- [CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts](https://mlsanity.com/paper/2604.10496/) (2026)
- [Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation](https://mlsanity.com/paper/2607.02968/) (2026)
- [CentriQ: Calibration-Free Quantization of Diffusion Transformers via Exact Mean Centering](https://mlsanity.com/paper/2610.06260/) (2026)

## Abstract (arXiv)

Diffusion Transformers (DiTs) have recently gained substantial attention in both industrial and academic fields for their superior visual generation capabilities, outperforming traditional diffusion models that use U-Net. However,the enhanced performance of DiTs also comes with high parameter counts and implementation costs, seriously restricting their use on resource-limited devices such as mobile phones. To address these challenges, we introduce the Hybrid Floating-point Quantization for DiT(HQ-DiT), an efficient post-training quantization method that utilizes 4-bit floating-point (FP) precision on both weights and activations for DiT inference. Compared to fixed-point quantization (e.g., INT8), FP quantization, complemented by our proposed clipping range selection mechanism, naturally aligns with the data distribution within DiT, resulting in a minimal quantization error. Furthermore, HQ-DiT also implements a universal identity mathematical transform to mitigate the serious quantization error caused by the outliers. The experimental results demonstrate that DiT can achieve extremely low-precision quantization (i.e., 4 bits) with negligible impact on performance. Our approach marks the first instance where both weights and activations in DiTs are quantized to just 4 bits, with only a 0.12 increase in sFID on ImageNet.