# ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models

arXiv: https://arxiv.org/abs/2406.10981 · HTML: https://mlsanity.com/paper/2406.10981/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 50 of its 59 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 50)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)

## Built on by (most cited first, 30 of 36)

- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models](https://mlsanity.com/paper/2504.12626/) (2025)
- [Video World Models with Long-term Spatial Memory](https://mlsanity.com/paper/2506.05284/) (2025)
- [Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition](https://mlsanity.com/paper/2506.17201/) (2025)
- [Progressive Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2410.08151/) (2024)
- [Long-Context State-Space Video World Models](https://mlsanity.com/paper/2505.20171/) (2025)
- [EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation](https://mlsanity.com/paper/2501.01895/) (2025)
- [Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model](https://mlsanity.com/paper/2511.23429/) (2025)
- [ControlAR: Controllable Image Generation with Autoregressive Models](https://mlsanity.com/paper/2410.02705/) (2024)
- [StreamDiT: Real-Time Streaming Text-to-Video Generation](https://mlsanity.com/paper/2507.03745/) (2025)
- [ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation](https://mlsanity.com/paper/2410.20502/) (2024)
- [Spatia: Video Generation with Updatable Spatial Memory](https://mlsanity.com/paper/2512.15716/) (2025)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes](https://mlsanity.com/paper/2409.04003/) (2024)
- [World Models for Learning Dexterous Hand-Object Interactions from Human Videos](https://mlsanity.com/paper/2512.13644/) (2025)
- [Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generation](https://mlsanity.com/paper/2508.03334/) (2025)
- [LoViC: Efficient Long Video Generation with Context Compression](https://mlsanity.com/paper/2507.12952/) (2025)
- [Streaming Autoregressive Video Generation via Diagonal Distillation](https://mlsanity.com/paper/2603.09488/) (2026)
- [Generative Pre-trained Autoregressive Diffusion Transformer](https://mlsanity.com/paper/2505.07344/) (2025)
- [MaskFlow: Discrete Flows For Flexible and Efficient Long Video Generation](https://mlsanity.com/paper/2502.11234/) (2025)
- [Physical Autoregressive Model for Robotic Manipulation without Action Pretraining](https://mlsanity.com/paper/2508.09822/) (2025)
- [From Virtual Games to Real-World Play](https://mlsanity.com/paper/2506.18901/) (2025)
- [MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion](https://mlsanity.com/paper/2410.07659/) (2024)
- [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [Block Cascading: Training Free Acceleration of Block-Causal Video Models](https://mlsanity.com/paper/2511.20426/) (2025)
- [DisCo: World Models with Discrete Camera Motion Control](https://mlsanity.com/paper/2606.07967/) (2026)
- [StarPose: 3D Human Pose Estimation via Spatial-Temporal Autoregressive Diffusion](https://mlsanity.com/paper/2508.02056/) (2025)
- [Generative Augmented Reality: Paradigms, Technologies, and Future Applications](https://mlsanity.com/paper/2511.16783/) (2025)

## Abstract (arXiv)

With the advance of diffusion models, today's video generation has achieved impressive quality. But generating temporal consistent long videos is still challenging. A majority of video diffusion models (VDMs) generate long videos in an autoregressive manner, i.e., generating subsequent clips conditioned on last frames of previous clip. However, existing approaches all involve bidirectional computations, which restricts the receptive context of each autoregression step, and results in the model lacking long-term dependencies. Inspired from the huge success of large language models (LLMs) and following GPT (generative pre-trained transformer), we bring causal (i.e., unidirectional) generation into VDMs, and use past frames as prompt to generate future frames. For Causal Generation, we introduce causal temporal attention into VDM, which forces each generated frame to depend on its previous frames. For Frame as Prompt, we inject the conditional frames by concatenating them with noisy frames (frames to be generated) along the temporal axis. Consequently, we present Video Diffusion GPT (ViD-GPT). Based on the two key designs, in each autoregression step, it is able to acquire long-term context from prompting frames concatenated by all previously generated frames. Additionally, we bring the kv-cache mechanism to VDMs, which eliminates the redundant computation from overlapped frames, significantly boosting the inference speed. Extensive experiments demonstrate that our ViD-GPT achieves state-of-the-art performance both quantitatively and qualitatively on long video generation. Code will be available at https://github.com/Dawn-LX/Causal-VideoGen.