mlsanityData to 7 Oct 2026

ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Models

arXiv 2311.18834 ↗First posted 30 Nov 2023cs.CVcited by 74

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present ARTboldsymbol·V, an efficient framework for auto-regressive video generation with diffusion models. Unlike existing methods that generate entire videos in one-shot, ARTboldsymbol·V generates a single frame at a time, conditioned on the previous ones. The framework offers three distinct advantages. First, it only learns simple continual motions between adjacent frames, therefore avoiding modeling complex long-range motions that require huge training data. Second, it preserves the high-fidelity generation ability of the pre-trained image diffusion models by making only minimal network modifications. Third, it can generate arbitrarily long videos conditioned on a variety of prompts such as text, image or their combinations, making it highly versatile and flexible. To combat the common drifting issue in AR models, we propose masked diffusion model which implicitly learns which information can be drawn from reference images rather than network predictions, in order to reduce the risk of generating inconsistent appearances that cause drifting. Moreover, we further enhance generation coherence by conditioning it on the initial frame, which typically contains minimal noise. This is particularly useful for long video generation. When trained for only two weeks on four GPUs, ARTboldsymbol·V already can generate videos with natural motions, rich details and a high level of aesthetic quality. Besides, it enables various appealing applications, e.g., composing a long video from multiple text prompts.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 62 of its 74 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

62 of 74 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  3. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  4. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  5. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  6. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  7. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation7 Sep
  8. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation1 Sep
  9. 2023ModelScope Text-to-Video Technical Report12 Aug
  10. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  11. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
7 more
  1. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  2. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  7. 2022Video Diffusion Models7 Apr

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
31 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  10. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  11. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  13. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  14. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  15. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  16. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  17. 2019Scaling Autoregressive Video Modelscited by 189
  18. 2017Video Generation From Textcited by 153
  19. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  20. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  21. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  22. 2016Video Pixel Networkscited by 108
  23. 2020Latent Video Transformercited by 106
  24. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  25. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  26. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  27. 2021Predicting Video with VQVAEcited by 61
  28. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  29. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44
  30. 2017Attentive Semantic Video Generation using Captionscited by 28
  31. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 34; 2026: 16.

74 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  2. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  3. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  4. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 209
  5. 2025Unified Video Action Modelcited by 195
  6. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  7. 2025Video World Models with Long-term Spatial Memorycited by 124
  8. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  9. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  10. 2025Mixture of Contexts for Long Video Generationcited by 90
  11. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
  12. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
38 more
  1. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 70
  2. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 68
  3. 2025Long-Context State-Space Video World Modelscited by 59
  4. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  5. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  6. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  7. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 44
  8. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generationcited by 41
  9. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  10. 2024Autoregressive Models in Vision: A Surveycited by 39
  11. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 34
  12. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Modelscited by 33
  13. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 33
  14. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  15. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  16. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  17. 2025Code2Video: A Code-centric Paradigm for Educational Video Generationcited by 23
  18. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  19. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  20. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Frameworkcited by 20
  21. 2025MAViS: A Multi-Agent Framework for Long-Sequence Video Storytellingcited by 19
  22. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 18
  23. 2025VideoMAR: Autoregressive Video Generatio with Continuous Tokenscited by 15
  24. 2025Towards One-step Causal Video Generation via Adversarial Self-Distillationcited by 15
  25. 2026Streaming Autoregressive Video Generation via Diagonal Distillationcited by 15
  26. 2024REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latentscited by 14
  27. 2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
  28. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Conceptscited by 12
  29. 2026MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Enginescited by 12
  30. 2025FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edgecited by 9
  31. 2025Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearningcited by 9
  32. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  33. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  34. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  35. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  36. 2025RFLAV: Rolling Flow matching for infinite Audio Video generationcited by 6
  37. 2025LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generationcited by 6
  38. 2025WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusioncited by 6

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion
  2. 2023MEVG: Multi-event Video Generation with Text-to-Video Models
  3. 2025EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  5. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  6. 2024Progressive Autoregressive Video Diffusion Models
  7. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
  8. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators