mlsanityData to 14 Oct 2026

Pyramidal Flow Matching for Efficient Video Generative Modeling

arXiv 2410.05954 ↗First posted 8 Oct 2024cs.CV cs.LGcited by 366

On the map: Diffusion models → Video generation

Abstract (arXiv)

Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https://pyramid-flow.github.io.

Key facts. Cited by 366 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 62 of its 75 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

62 of 75 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Diffusion Models Are Real-Time Game Engines27 Aug
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  3. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback29 May
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  5. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  7. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation9 Oct
  8. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  9. 2023ModelScope Text-to-Video Technical Report12 Aug
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  12. 2022Video Diffusion Models7 Apr

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2023Segment Anythingcited by 8,234
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2022Flow Matching for Generative Modelingcited by 5,702
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  10. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  11. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  14. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  15. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  16. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  17. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  18. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  19. 2024Genie: Generative Interactive Environmentscited by 747
  20. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  21. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  22. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  23. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  24. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  25. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  26. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  27. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  28. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  29. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 118
  30. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 106
  31. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  32. 2024PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Acceleratorcited by 97
  33. 2022Poisson Flow Generative Modelscited by 95
  34. 2023Relay Diffusion: Unifying diffusion process across resolutions for image synthesiscited by 79
  35. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 55
  36. 2023Trans-Dimensional Generative Modeling via Jump Diffusion Modelscited by 35
  37. 2022f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformationcited by 32
  38. 2022Dimensionality-Varying Diffusion Processcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 176; 2026: 172.

366 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  3. 2024LTX-Video: Realtime Video Latent Diffusioncited by 581
  4. 2026World Action Models are Zero-shot Policiescited by 465
  5. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  6. 2025LongLive: Real-time Interactive Long Video Generationcited by 265
  7. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  8. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  9. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 186
  10. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  11. 2025History-Guided Video Diffusioncited by 171
  12. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
38 more
  1. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  2. 2025Video World Models with Long-term Spatial Memorycited by 136
  3. 2025Test-Time Training Done Rightcited by 130
  4. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
  5. 2025Aether: Geometric-Aware Unified World Modelingcited by 97
  6. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 97
  7. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 96
  8. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  9. 2025Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rolloutcited by 86
  10. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
  11. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 85
  12. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 76
  13. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  14. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  15. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  16. 2025Long-Context State-Space Video World Modelscited by 64
  17. 2025SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restorationcited by 62
  18. 2026Context Forcing: Consistent Autoregressive Video Generation with Long Contextcited by 59
  19. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 56
  20. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 52
  21. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  22. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
  23. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  24. 2025MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narrativescited by 45
  25. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 42
  26. 2025Video-T1: Test-Time Scaling for Video Generationcited by 40
  27. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 40
  28. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
  29. 2024Autoregressive Models in Vision: A Surveycited by 39
  30. 2025Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectoriescited by 39
  31. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Modelscited by 38
  32. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 35
  33. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  34. 2025FlashVideo: Flowing Fidelity to Detail for Efficient High-Resolution Video Generationcited by 32
  35. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 32
  36. 2026LoL: Longer than Longer, Scaling Video Generation to Hourcited by 32
  37. 2026Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusioncited by 32
  38. 2024FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusioncited by 30

50 most cited of 366 All 366 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FlowLong: Inference-time Long Video Generation via Manifold-constrained Tweedie Matching
  2. 2026RhymeFlow: Training-Free Acceleration for Video Generation with Asynchronous Denoising Flow Scheduling
  3. 2025Iterative Flow Matching: Path Correction and Gradual Refinement for Enhanced Generative Modeling
  4. 2025STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows
  5. 2024Ultra-High-Resolution Image Synthesis with Pyramid Diffusion Model
  6. 2026Transition Matching Distillation for Fast Video Generation
  7. 2026DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation
  8. 2026Dynamic Video Generation: Shaping Video Generation Across Time and Space