mlsanityData to 7 Oct 2026

Real-Time Video Generation with Pyramid Attention Broadcast

arXiv 2408.12588 ↗First posted 22 Aug 2024cs.CV cs.DCcited by 140

Not on the map. Why?

Cited by 116 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates up to 10.5x speedup across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 52 of its 74 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

52 of 74 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
38 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  9. 2023Adversarial Diffusion Distillationcited by 741
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  11. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  12. 2023ModelScope Text-to-Video Technical Reportcited by 651
  13. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  14. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  15. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  16. 2016Generating Videos with Scene Dynamicscited by 376
  17. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  18. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  19. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  20. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  21. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  22. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  23. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  24. 2021Gotta Go Fast When Generating Data with Score-Based Modelscited by 218
  25. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  26. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  27. 2019Scaling Autoregressive Video Modelscited by 189
  28. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 169
  29. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 149
  30. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138
  31. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  32. 2020Latent Video Transformercited by 106
  33. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  34. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 91
  35. 2023Temporal Dynamic Quantization for Diffusion Modelscited by 81
  36. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Accelerationcited by 70
  37. 2019Lower Dimensional Kernels for Video Discriminatorscited by 37
  38. 2017Moonshine: Distilling with Cheap Convolutionscited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 51; 2026: 70.

140 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  3. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  4. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 245
  5. 2024Accelerating Diffusion Transformers with Token-wise Feature Cachingcited by 143
  6. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  7. 2025From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeerscited by 124
  8. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  9. 2025Training Agents Inside of Scalable World Modelscited by 122
  10. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Qualitycited by 115
  11. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  12. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
38 more
  1. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  2. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 55
  3. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  4. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 44
  5. 2025Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Cachingcited by 39
  6. 2025MagCache: Fast Video Generation with Magnitude-Aware Cachecited by 36
  7. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  8. 2025DiCache: Let Diffusion Model Determine Its Own Cachecited by 34
  9. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 32
  10. 2024UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Drivingcited by 27
  11. 2026Transition Matching Distillation for Fast Video Generationcited by 27
  12. 2024Rethinking Token-wise Feature Caching: Accelerating Diffusion Transformers with Dual Feature Cachingcited by 26
  13. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tilecited by 26
  14. 2025Training-free Diffusion Acceleration with Bottleneck Samplingcited by 25
  15. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  16. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 23
  17. 2025Forecast then Calibrate: Feature Caching as ODE for Efficient Diffusion Transformerscited by 23
  18. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  19. 2025Dual-Expert Consistency Model for Efficient and High-Quality Video Generationcited by 21
  20. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  21. 2025Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesiscited by 20
  22. 2024SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformerscited by 19
  23. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  24. 2025Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Cachingcited by 17
  25. 2025FreqCa: Accelerating Diffusion Models via Frequency-Aware Cachingcited by 16
  26. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 15
  27. 2024REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latentscited by 14
  28. 2025Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Modelscited by 14
  29. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 14
  30. 2025UniCP: A Unified Caching and Pruning Framework for Efficient Video Generationcited by 13
  31. 2025FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Computecited by 12
  32. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  33. 2024Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraintscited by 11
  34. 2025AB-Cache: Training-Free Acceleration of Diffusion Models via Adams-Bashforth Cached Feature Reusecited by 11
  35. 2025Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformerscited by 11
  36. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  37. 2025KlingAvatar 2.0 Technical Reportcited by 11
  38. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10

50 most cited of 140 All 140 (JSON)