mlsanityData to 14 Oct 2026

Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models

arXiv 2405.05252 ↗First posted 8 May 2024cs.CV cs.AI cs.LGcited by 38

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion Models (DMs) have exhibited superior performance in generating high-quality and diverse images. However, this exceptional performance comes at the cost of expensive architectural design, particularly due to the attention module heavily used in leading models. Existing works mainly adopt a retraining process to enhance DM efficiency. This is computationally expensive and not very scalable. To this end, we introduce the Attention-driven Training-free Efficient Diffusion Model (AT-EDM) framework that leverages attention maps to perform run-time pruning of redundant tokens, without the need for any retraining. Specifically, for single-denoising-step pruning, we develop a novel ranking algorithm, Generalized Weighted Page Rank (G-WPR), to identify redundant tokens, and a similarity-based recovery method to restore tokens for the convolution operation. In addition, we propose a Denoising-Steps-Aware Pruning (DSAP) approach to adjust the pruning budget across different denoising timesteps for better generation quality. Extensive evaluations show that AT-EDM performs favorably against prior art in terms of efficiency (e.g., 38.8% FLOPs saving and up to 1.53x speed-up over Stable Diffusion XL) while maintaining nearly the same FID and CLIP scores as the full model. Project webpage: https://atedm.github.io.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 37 of its 42 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

37 of 42 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  2. 2023Denoising Diffusion Step-aware Models5 Oct
  3. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration19 Sep
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  5. 2023OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models15 Jun
  6. 2023Structural Pruning for Diffusion Models18 May
  7. 2023Token Merging for Fast Stable Diffusion30 Mar
  8. 2023Consistency Models2 Mar
  9. 2022Diffusion Probabilistic Model Made Slim27 Nov
  10. 2022On Distillation of Guided Diffusion Models6 Oct
  11. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  12. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
7 more
  1. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  2. 2021Score-based Generative Modeling in Latent Space10 Jun
  3. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  4. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  6. 2020Denoising Diffusion Implicit Models6 Oct
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 18

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  9. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  10. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  11. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  12. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
6 more
  1. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  2. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  3. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  4. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  5. 2020Neuron Merging: Compensating for Pruned Neuronscited by 24
  6. 2021RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networkscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 16; 2026: 14.

38 papers.

Sort:

By area:

  1. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 269
  2. 2024FORA: Fast-Forward Caching in Diffusion Transformer Accelerationcited by 137
  3. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Qualitycited by 122
  4. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  5. 2025Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Cachingcited by 45
  6. 2024LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexitycited by 43
  7. 2025MagCache: Fast Video Generation with Magnitude-Aware Cachecited by 41
  8. 2025FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruningcited by 34
  9. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  10. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformerscited by 22
  11. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21
  12. 2025Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Accelerationcited by 17
26 more
  1. 2025CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Modelscited by 16
  2. 2025SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skippingcited by 11
  3. 2026PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generationcited by 8
  4. 2025RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancycited by 7
  5. 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformercited by 6
  6. 2025Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deploymentcited by 5
  7. 2025Attention in Diffusion Model: A Surveycited by 4
  8. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 4
  9. 2025Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generationcited by 3
  10. 2026Attention Sinks in Diffusion Transformers: A Causal Analysiscited by 3
  11. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  12. 2024Time Step Generating: A Universal Synthesized Deepfake Image Detectorcited by 2
  13. 2026Shiva-DiT: Residual-Based Differentiable Top-k Selection for Efficient Diffusion Transformerscited by 2
  14. 2026CHAI: CacHe Attention Inference for text2videocited by 2
  15. 2025SlimDiff: Training-Free, Activation-Guided Hands-free Slimming of Diffusion Modelscited by 1
  16. 2025Generative Augmented Reality: Paradigms, Technologies, and Future Applicationscited by 1
  17. 2026TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolutioncited by 1
  18. 2025OptiPrune: Boosting Prompt-Image Consistency with Attention-Guided Noise and Dynamic Token Selectioncited by 0
  19. 2026RemEdit: Efficient Diffusion Editing with Riemannian Geometrycited by 0
  20. 2026SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformercited by 0
  21. 2026Turbo4DGen: Ultra-Fast Acceleration for 4D Generationcited by 0
  22. 2026Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generationcited by 0
  23. 2026HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editingcited by 0
  24. 2026Importance-Aware OBS Pruning for Diffusion Modelscited by 0
  25. 2026MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generationcited by 0
  26. 2026Heuristics are Not Enough: Calibrated and Corrected Caching for Video Diffusion Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learnable Sparsity for Vision Generative Models
  2. 2023Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models
  3. 2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization
  4. 2023Diffusion Models Without Attention
  5. 2025ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
  6. 2025Glance: Accelerating Diffusion Models with 1 Sample
  7. 2026Denoising Diffusion Generative Models Secretly Calculate Attentions
  8. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models