mlsanityData to 14 Oct 2026

SparseDM: Toward Sparse Efficient Diffusion Models

arXiv 2404.10445 ↗First posted 16 Apr 2024cs.LG cs.AIcited by 18

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models represent a powerful family of generative models widely used for image and video generation. However, the time-consuming deployment, long inference time, and requirements on large memory hinder their applications on resource constrained devices. In this paper, we propose a method based on the improved Straight-Through Estimator to improve the deployment efficiency of diffusion models. Specifically, we add sparse masks to the Convolution and Linear layers in a pre-trained diffusion model, then transfer learn the sparse model during the fine-tuning stage and turn on the sparse masks during inference. Experimental results on a Transformer and UNet-based diffusion models demonstrate that our method reduces MACs by 50% while maintaining FID. Sparse models are accelerated by approximately 1.2x on the GPU. Under other MACs conditions, the FID is also lower than 1 compared to other methods.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 32 of its 49 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

32 of 49 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Module9 Nov
  2. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics20 Oct
  3. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  4. 2023Structural Pruning for Diffusion Models18 May
  5. 2023Token Merging for Fast Stable Diffusion30 Mar
  6. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models9 Feb
  7. 2022Scalable Diffusion Models with Transformers19 Dec
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  9. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  12. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
3 more
  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  2. 2020Denoising Diffusion Implicit Models6 Oct
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 17

  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  6. 2021Accelerating Sparse Deep Neural Networkscited by 223
  7. 2023DepGraph: Towards Any Structural Pruningcited by 203
  8. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  9. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  10. 2022Learning Best Combination for Efficient N:M Sparsitycited by 32
  11. 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesiscited by 30
  12. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
5 more
  1. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 19
  2. 2023Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approachcited by 12
  3. 2022Learning Moving-Object Tracking with FMCW LiDARcited by 5
  4. 2022Toward Efficient Automated Feature Engineeringcited by 2
  5. 2021SenseMag: Enabling Low-Cost Traffic Monitoring using Non-invasive Magnetic Sensingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024Dynamic Diffusion Transformercited by 62
  2. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 45
  3. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 37
  4. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  5. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 20
  6. 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Accelerationcited by 17
  7. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  8. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 7
  9. 2026Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generationcited by 7
  10. 2025Pruning for Sparse Diffusion Models based on Gradient Flowcited by 5
  11. 2026DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformerscited by 5
  12. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 4
6 more
  1. 2025SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsitycited by 3
  2. 2026Token Pruning for In-Context Generation in Diffusion Transformerscited by 2
  3. 2025DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generationcited by 1
  4. 2025Sparse-to-Sparse Training of Diffusion Modelscited by 1
  5. 20252ndMatch: Finetuning Pruned Diffusion Models via Second-Order Jacobian Matchingcited by 0
  6. 2026Importance-Aware OBS Pruning for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Models
  2. 2026Reviving ConvNeXt for Efficient Convolutional Diffusion Models
  3. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  4. 2025Toward Lightweight and Fast Decoders for Diffusion Models in Image and Video Generation
  5. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas
  6. 2025Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers
  7. 2025Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Finance
  8. 2026SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference