mlsanityData to 7 Oct 2026

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

arXiv 2411.05007 ↗First posted 7 Nov 2024cs.CV cs.LGcited by 161

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use a high-precision, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while a low-bit quantized branch handles the residuals. This process eases the quantization on both sides. However, naively running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-Sigma, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5×, achieving 3.0× speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve a 3.1× speedup compared to the W4A16 model using NVFP4 precision.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 80 of its 102 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

80 of 102 references.

Closest ancestors · 38

Same direction, newest first

  1. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models14 Oct
  2. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers14 Oct
  3. 2024AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising11 Jun
  4. 2024BitsFusion: 1.99 bits Weight Quantization of Diffusion Model6 Jun
  5. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation4 Jun
  6. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching3 Jun
  7. 2024HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization30 May
  8. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization28 May
  9. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformers25 May
  10. 2024Improved Distribution Matching Distillation for Fast Image Synthesis23 May
  11. 2024Distilling Diffusion Models into Conditional GANs9 May
  12. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation7 Mar
26 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  2. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models29 Feb
  3. 2024QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning6 Feb
  4. 2024EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models9 Jan
  5. 2023Efficient Quantization Strategies for Latent Diffusion Models9 Dec
  6. 2023DeepCache: Accelerating Diffusion Models for Free1 Dec
  7. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  8. 2023Adversarial Diffusion Distillation28 Nov
  9. 2023TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models27 Nov
  10. 2023Post-training Quantization for Text-to-Image Diffusion Models with Progressive Calibration and Activation Relaxing10 Nov
  11. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  12. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models5 Oct
  13. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  14. 2023Towards Accurate Post-training Quantization for Diffusion Models30 May
  15. 2023PTQD: Accurate Post-Training Quantization for Diffusion Models18 May
  16. 2023Consistency Models2 Mar
  17. 2023Q-Diffusion: Quantizing Diffusion Models8 Feb
  18. 2022Scalable Diffusion Models with Transformers19 Dec
  19. 2022Post-training Quantization on Diffusion Models28 Nov
  20. 2022On Distillation of Guided Diffusion Models6 Oct
  21. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  22. 2022gDDIM: Generalized denoising diffusion implicit models11 Jun
  23. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  24. 2022Fast Sampling of Diffusion Models with Exponential Integrator29 Apr
  25. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  26. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
30 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  8. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  10. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  11. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  12. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  13. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  14. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  15. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  16. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  17. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  18. 2022Language model compression with weighted low-rank factorizationcited by 192
  19. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  20. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  21. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  22. 2023Microscaling Data Formats for Deep Learningcited by 132
  23. 2024One-Step Image Translation with Text-to-Image Modelscited by 111
  24. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  25. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  26. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  27. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  28. 2022Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Modelscited by 44
  29. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
  30. 2024Condition-Aware Neural Network for Controlled Image Generationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 75; 2026: 81.

161 papers.

Sort:

By area:

  1. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  3. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  4. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  5. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 85
  6. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  7. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  8. 2025DiCache: Let Diffusion Model Determine Its Own Cachecited by 34
  9. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 32
  10. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  11. 20241.58-bit FLUXcited by 31
  12. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
38 more
  1. 2024Rethinking Token-wise Feature Caching: Accelerating Diffusion Transformers with Dual Feature Cachingcited by 26
  2. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  3. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 25
  4. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  5. 2025HiFlow: Training-free High-Resolution Image Generation with Flow-Aligned Guidancecited by 21
  6. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 21
  7. 2025Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervalscited by 21
  8. 2025Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformerscited by 20
  9. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 20
  10. 2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 19
  11. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  12. 2026Flow caching for autoregressive video generationcited by 19
  13. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 18
  14. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  15. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  16. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Modelscited by 15
  17. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 15
  18. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  19. 2025DVD-Quant: Data-free Video Diffusion Transformers Quantizationcited by 14
  20. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 14
  21. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 12
  22. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  23. 2025Scaling Group Inference for Diverse and High-Quality Generationcited by 11
  24. 2025ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusioncited by 11
  25. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 10
  26. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  27. 2025Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMscited by 9
  28. 2025Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformerscited by 9
  29. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  30. 2025MosaicDiff: Training-free Structural Pruning for Diffusion Model Acceleration Reflecting Pretraining Dynamicscited by 9
  31. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 9
  32. 2026Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formatscited by 8
  33. 2025SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperationcited by 7
  34. 2025DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacecited by 7
  35. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformercited by 7
  36. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  37. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bitscited by 6
  38. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models
  2. 2025DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization
  3. 2024MPQ-Diff: Mixed Precision Quantization for Diffusion Models
  4. 2025AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models
  5. 2025AQUATIC-Diff: Additive Quantization for Truly Tiny Compressed Diffusion Models
  6. 2026FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models
  7. 2024PQD: Post-training Quantization for Efficient Diffusion Models
  8. 2024QNCD: Quantization Noise Correction for Diffusion Models