mlsanityData to 7 Oct 2026

From Slow Bidirectional to Fast Autoregressive Video Diffusion Models

arXiv 2412.07772 ↗First posted 10 Dec 2024cs.CVcited by 401

On the map: Diffusion models → Video generation

Abstract (arXiv)

Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner.

Key facts. Cited by 401 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 99 of its 113 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

99 of 113 references.

Closest ancestors · 29

Same direction, newest first

  1. 2024GameGen-X: Interactive Open-world Game Video Generation1 Nov
  2. 2024Progressive Autoregressive Video Diffusion Models10 Oct
  3. 2024Pyramidal Flow Matching for Efficient Video Generative Modeling8 Oct
  4. 2024OSV: One Step is Enough for High-Quality Image to Video Generation17 Sep
  5. 2024Diffusion Models Are Real-Time Game Engines27 Aug
  6. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  7. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence23 Jul
  8. 2024Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models11 Jul
  9. 2024Video-Infinity: Distributed Long Video Generation24 Jun
  10. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models16 Jun
  11. 2024SF-V: Single Forward Video Generation Model6 Jun
  12. 2024Streaming Video Diffusion: Online Video Editing with Diffusion Models30 May
17 more
  1. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback29 May
  2. 2024Looking Backward: Streaming Video-to-Video Translation with Feature Banks24 May
  3. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Training19 May
  4. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  5. 2024AnimateDiff-Lightning: Cross-Model Diffusion Distillation19 Mar
  6. 2023VideoLCM: Video Latent Consistency Model14 Dec
  7. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Models30 Nov
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  9. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  10. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  11. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  12. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  13. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  14. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  15. 2022Scalable Adaptive Computation for Iterative Generation22 Dec
  16. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  17. 2022Video Diffusion Models7 Apr

Other references · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2023Consistency Modelscited by 1,778
  7. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  9. 2021Variational Diffusion Modelscited by 1,145
  10. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  13. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  14. 2023Adversarial Diffusion Distillationcited by 741
  15. 2022On Distillation of Guided Diffusion Modelscited by 721
  16. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  18. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  19. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  20. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  21. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  22. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  23. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  24. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  25. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  26. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  27. 2023Improved Techniques for Training Consistency Modelscited by 430
  28. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  29. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  30. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  31. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  32. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  33. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  34. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  35. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  36. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  37. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  38. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 112; 2026: 289.

401 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  3. 2026World Action Models are Zero-shot Policiescited by 411
  4. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  5. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  6. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 209
  7. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  8. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  9. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  10. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  11. 2026Advancing Open-source World Modelscited by 144
  12. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
38 more
  1. 2025Video World Models with Long-term Spatial Memorycited by 124
  2. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 111
  3. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  4. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  5. 2025Mixture of Contexts for Long Video Generationcited by 90
  6. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  7. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 86
  8. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  9. 2025Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rolloutcited by 78
  10. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  11. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 75
  12. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 72
  13. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  14. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 68
  15. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  16. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  17. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  18. 2025Long-Context State-Space Video World Modelscited by 59
  19. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  20. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  21. 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 51
  22. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
  23. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 44
  24. 2025StreamDiT: Real-Time Streaming Text-to-Video Generationcited by 43
  25. 2025MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narrativescited by 40
  26. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 39
  27. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  28. 2026Infinite Worlds with Versatile Interactionscited by 37
  29. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  30. 2026MIND: Benchmarking Memory Consistency and Action Control in World Modelscited by 31
  31. 2026LoL: Longer than Longer, Scaling Video Generation to Hourcited by 29
  32. 2026Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusioncited by 28
  33. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 27
  34. 2026Transition Matching Distillation for Fast Video Generationcited by 27
  35. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 25
  36. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  37. 2025A Survey of Interactive Generative Videocited by 23
  38. 2026AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillationcited by 23

50 most cited of 401 All 401 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation
  2. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion
  3. 2025Generative Pre-trained Autoregressive Diffusion Transformer
  4. 2026Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
  5. 2026S2PD: Serial-to-Parallel Diffusion for Physically and Logically Consistent Video Generation
  6. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization
  7. 2024Accelerating Video Diffusion Models via Distribution Matching
  8. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation