mlsanityData to 14 Oct 2026

Adaptive Caching for Faster Video Generation with Diffusion Transformers

arXiv 2411.02397 ↗First posted 4 Nov 2024cs.CVcited by 108

On the map: Diffusion models → Video generation

Abstract (arXiv)

Generating temporally-consistent high-fidelity videos can be computationally expensive, especially over longer temporal spans. More-recent Diffusion Transformers (DiTs) -- despite making significant headway in this context -- have only heightened such challenges as they rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. In this paper, we introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache), which is motivated by the fact that "not all videos are created equal": meaning, some videos require fewer denoising steps to attain a reasonable quality than others. Building on this, we not only cache computations through the diffusion process, but also devise a caching schedule tailored to each video generation, maximizing the quality-latency trade-off. We further introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, essentially controlling the compute allocation based on motion content. Altogether, our plug-and-play contributions grant significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video generation) without sacrificing the generation quality, across multiple video DiT baselines.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 103 of its 121 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

103 of 121 references.

Closest ancestors · 28

Same direction, newest first

  1. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scale26 Oct
  2. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality25 Oct
  3. 2024Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation27 Aug
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  5. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generation31 Jul
  6. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence23 Jul
  7. 2024Video-Infinity: Distributed Long Video Generation24 Jun
  8. 2024Compositional Video Generation as Flow Equalization10 Jun
  9. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation4 Jun
  10. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation2 May
  11. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  12. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
16 more
  1. 2024Object-Centric Diffusion for Efficient Video Editing11 Jan
  2. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  3. 2023VidToMe: Video Token Merging for Zero-Shot Video Editing17 Dec
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  5. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  6. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  7. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  8. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  9. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  10. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling22 May
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  12. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  13. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  14. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  15. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  16. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 75

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2022Generative Adversarial Networkscited by 5,627
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
38 more
  1. 2023Consistency Modelscited by 1,822
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  7. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  9. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  10. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  12. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  13. 2023Adversarial Diffusion Distillationcited by 759
  14. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  15. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  16. 2022On Distillation of Guided Diffusion Modelscited by 725
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  18. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  19. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  20. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  21. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  22. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  23. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  24. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  25. 2022Blended Latent Diffusioncited by 432
  26. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  27. 2016Generating Videos with Scene Dynamicscited by 376
  28. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  29. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  30. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  31. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  32. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 278
  33. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  34. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  35. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  36. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  37. 2023Token Merging for Fast Stable Diffusioncited by 224
  38. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 33; 2026: 71.

108 papers.

Sort:

By area:

  1. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 269
  2. 2025From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeerscited by 136
  3. 2025Yume: An Interactive World Generation Modelcited by 81
  4. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  5. 2025DiCache: Let Diffusion Model Determine Its Own Cachecited by 42
  6. 2025MagCache: Fast Video Generation with Magnitude-Aware Cachecited by 41
  7. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  8. 2026Flow caching for autoregressive video generationcited by 25
  9. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 24
  10. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  11. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  12. 2025QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generationcited by 21
38 more
  1. 2025OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Modelscited by 18
  2. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 16
  3. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 16
  4. 2026OmniRoam: World Wandering via Long-Horizon Panoramic Video Generationcited by 16
  5. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  6. 2025Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Modelscited by 15
  7. 2025DVD-Quant: Data-free Video Diffusion Transformers Quantizationcited by 15
  8. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 14
  9. 2025S2Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillationcited by 13
  10. 2024Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraintscited by 12
  11. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  12. 2025ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusioncited by 12
  13. 2025Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformerscited by 11
  14. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  15. 2026SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Modelscited by 11
  16. 2025TaoCache: Structure-Maintained Video Generation Accelerationcited by 10
  17. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  18. 2025HERO: Hierarchical Extrapolation and Refresh for Efficient World Modelscited by 9
  19. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 8
  20. 2026Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformerscited by 8
  21. 2026Elastic Diffusion Transformercited by 8
  22. 2026Denoising as Path Planning: Training-Free Acceleration of Diffusion Models with DPCachecited by 8
  23. 2026PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generationcited by 8
  24. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Ratecited by 7
  25. 2025RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancycited by 7
  26. 2026SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformercited by 7
  27. 2026DisCa: Accelerating Video Diffusion Transformers with Distillation-Compatible Learnable Feature Cachingcited by 7
  28. 2026SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Cachingcited by 7
  29. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 7
  30. 2025Foresight: Adaptive Layer Reuse for Accelerated and High-Quality Text-to-Video Generationcited by 6
  31. 2025SuperGen: An Efficient Ultra-high-resolution Video Generation System with Sketching and Tilingcited by 6
  32. 2025FreeViS: Training-free Video Stylization with Inconsistent Referencescited by 6
  33. 2026WorldCache: Accelerating World Models for Free via Heterogeneous Token Cachingcited by 6
  34. 2026Motion-Aware Caching for Efficient Autoregressive Video Generationcited by 6
  35. 2026DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generationcited by 6
  36. 2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generationcited by 5
  37. 2026LESA: Learnable Stage-Aware Predictors for Diffusion Model Accelerationcited by 5
  38. 2026Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolationcited by 5

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Caching
  2. 2025Adaptive Hybrid Caching for Efficient Text-to-Video Diffusion Model Acceleration
  3. 2026MORCA: Offline-to-Online Reinforcement Learning for Adaptive Cache Reuse in Video Diffusion Acceleration
  4. 2026Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep
  5. 2026Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers
  6. 2025LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation
  7. 2026From Local Mismatch to Global Impact: Optimizing Cache Reuse Policy for Efficient Diffusion
  8. 2025LeMiCa: Lexicographic Minimax Path Caching for Efficient Diffusion-Based Video Generation