# Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning

arXiv: https://arxiv.org/abs/2505.21591 · HTML: https://mlsanity.com/paper/2505.21591/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Q-Diffusion: Quantizing Diffusion Models](https://mlsanity.com/paper/2302.04304/) (2023)
- [Post-training Quantization on Diffusion Models](https://mlsanity.com/paper/2211.15736/) (2022)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)
- [PTQD: Accurate Post-Training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.10657/) (2023)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [Temporal Dynamic Quantization for Diffusion Models](https://mlsanity.com/paper/2306.02316/) (2023)
- [EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models](https://mlsanity.com/paper/2310.03270/) (2023)
- [LLM-FP4: 4-Bit Floating-Point Quantized Transformers](https://mlsanity.com/paper/2310.16836/) (2023)
- [An empirical study of LLaMA3 quantization: from LLMs to MLLMs](https://mlsanity.com/paper/2404.14047/) (2024)
- [Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later](https://mlsanity.com/paper/2303.02490/) (2023)
- [QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning](https://mlsanity.com/paper/2402.03666/) (2024)
- [Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models](https://mlsanity.com/paper/2403.03432/) (2024)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)

## Built on by (most cited first, 6 of 6)

- [Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers](https://mlsanity.com/paper/2506.03065/) (2025)
- [RegionE: Adaptive Region-Aware Generation for Efficient Image Editing](https://mlsanity.com/paper/2510.25590/) (2025)
- [DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers](https://mlsanity.com/paper/2605.16732/) (2026)
- [LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation](https://mlsanity.com/paper/2510.05367/) (2025)
- [2ndMatch: Finetuning Pruned Diffusion Models via Second-Order Jacobian Matching](https://mlsanity.com/paper/2506.05398/) (2025)
- [LoRA-Diffusion: Parameter-Efficient Fine-Tuning via Low-Rank Trajectory Decomposition](https://mlsanity.com/paper/2608.12328/) (2026)

## Abstract (arXiv)

Model quantization reduces the bit-width of weights and activations, improving memory efficiency and inference speed in diffusion models. However, achieving 4-bit quantization remains challenging. Existing methods, primarily based on integer quantization and post-training quantization fine-tuning, struggle with inconsistent performance. Inspired by the success of floating-point (FP) quantization in large language models, we explore low-bit FP quantization for diffusion models and identify key challenges: the failure of signed FP quantization to handle asymmetric activation distributions, the insufficient consideration of temporal complexity in the denoising process during fine-tuning, and the misalignment between fine-tuning loss and quantization error. To address these challenges, we propose the mixup-sign floating-point quantization (MSFP) framework, first introducing unsigned FP quantization in model quantization, along with timestep-aware LoRA (TALoRA) and denoising-factor loss alignment (DFA), which ensure precise and stable fine-tuning. Extensive experiments show that we are the first to achieve superior performance in 4-bit FP quantization for diffusion models, outperforming existing PTQ fine-tuning methods in 4-bit INT quantization.