# QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification

arXiv: https://arxiv.org/abs/2509.23681 · HTML: https://mlsanity.com/paper/2509.23681/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 34 of its 65 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 34)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025)
- [Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](https://mlsanity.com/paper/2405.08748/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [MoBA: Mixture of Block Attention for Long-Context LLMs](https://mlsanity.com/paper/2502.13189/) (2025)
- [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://mlsanity.com/paper/2411.05007/) (2024)
- [SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://mlsanity.com/paper/2410.02367/) (2024)
- [FlatQuant: Flatness Matters for LLM Quantization](https://mlsanity.com/paper/2410.09426/) (2024)
- [Fast Video Generation with Sliding Tile Attention](https://mlsanity.com/paper/2502.04507/) (2025)
- [SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs](https://mlsanity.com/paper/2410.13276/) (2024)
- [SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference](https://mlsanity.com/paper/2502.18137/) (2025)
- [ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation](https://mlsanity.com/paper/2406.02540/) (2024)
- [SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization](https://mlsanity.com/paper/2411.10958/) (2024)
- [Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers](https://mlsanity.com/paper/2406.17343/) (2024)
- [PTQ4DiT: Post-training Quantization for Diffusion Transformers](https://mlsanity.com/paper/2405.16005/) (2024)
- [EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models](https://mlsanity.com/paper/2310.03270/) (2023)
- [AffineQuant: Affine Transformation Quantization for Large Language Models](https://mlsanity.com/paper/2403.12544/) (2024)
- [SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training](https://mlsanity.com/paper/2505.11594/) (2025)
- [Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers](https://mlsanity.com/paper/2406.16747/) (2024)
- [Training-Free Efficient Video Generation via Dynamic Token Carving](https://mlsanity.com/paper/2505.16864/) (2025)
- [From Sora What We Can See: A Survey of Text-to-Video Generation](https://mlsanity.com/paper/2405.10674/) (2024)
- [Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers](https://mlsanity.com/paper/2505.14687/) (2025)

## Built on by (most cited first, 7 of 7)

- [Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation](https://mlsanity.com/paper/2605.16003/) (2026)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [WorldCache: Accelerating World Models for Free via Heterogeneous Token Caching](https://mlsanity.com/paper/2603.06331/) (2026)
- [Fast-SAM3D: 3Dfy Anything in Images but Faster](https://mlsanity.com/paper/2602.05293/) (2026)
- [LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration](https://mlsanity.com/paper/2608.12032/) (2026)
- [SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention](https://mlsanity.com/paper/2608.12780/) (2026)
- [MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers](https://mlsanity.com/paper/2610.06801/) (2026)

## Abstract (arXiv)

Diffusion transformers exhibit remarkable video generation capability, yet their prohibitive computational and memory costs hinder practical deployment. Model quantization and attention sparsification are two promising directions for compression, but each alone suffers severe performance degradation under aggressive compression. Combining them promises compounded efficiency gains, but naive integration is ineffective. The sparsity-induced information loss exacerbates quantization noise, leading to amplified attention shifts. To address this, we propose QuantSparse, a unified framework that integrates model quantization with attention sparsification. Specifically, we introduce textitMulti-Scale Salient Attention Distillation, which leverages both global structural guidance and local salient supervision to mitigate quantization-induced bias. In addition, we develop textitSecond-Order Sparse Attention Reparameterization, which exploits the temporal stability of second-order residuals to efficiently recover information lost under sparsity. Experiments on HunyuanVideo-13B demonstrate that QuantSparse achieves 20.88 PSNR, substantially outperforming the state-of-the-art quantization baseline Q-VDiT (16.85 PSNR), while simultaneously delivering a 3.68× reduction in storage and 1.88× acceleration in end-to-end inference. Our code will be released in https://github.com/wlfeng0509/QuantSparse.