# Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation

arXiv: https://arxiv.org/abs/2309.03549 · HTML: https://mlsanity.com/paper/2309.03549/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 63 of its 69 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)

## Built on by (most cited first, 30 of 50)

- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024)
- [Make Pixels Dance: High-Dynamic Video Generation](https://mlsanity.com/paper/2311.10982/) (2023)
- [Boximator: Generating Rich and Controllable Motions for Video Synthesis](https://mlsanity.com/paper/2402.01566/) (2024)
- [FreeInit: Bridging Initialization Gap in Video Diffusion Models](https://mlsanity.com/paper/2312.07537/) (2023)
- [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://mlsanity.com/paper/2312.12490/) (2023)
- [ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Models](https://mlsanity.com/paper/2311.18834/) (2023)
- [On the Content Bias in Fr'echet Video Distance](https://mlsanity.com/paper/2404.12391/) (2024)
- [StreamDiT: Real-Time Streaming Text-to-Video Generation](https://mlsanity.com/paper/2507.03745/) (2025)
- [MEVG: Multi-event Video Generation with Text-to-Video Models](https://mlsanity.com/paper/2312.04086/) (2023)
- [A Survey on Long Video Generation: Challenges, Methods, and Prospects](https://mlsanity.com/paper/2403.16407/) (2024)
- [Mind the Time: Temporally-Controlled Multi-Event Video Generation](https://mlsanity.com/paper/2412.05263/) (2024)
- [Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment](https://mlsanity.com/paper/2403.11956/) (2024)
- [DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Framework](https://mlsanity.com/paper/2408.11788/) (2024)
- [MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation](https://mlsanity.com/paper/2311.18829/) (2023)
- [Quality Assessment in the Era of Large Models: A Survey](https://mlsanity.com/paper/2409.00031/) (2024)
- [MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing](https://mlsanity.com/paper/2311.17338/) (2023)
- [BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models](https://mlsanity.com/paper/2312.02813/) (2023)
- [DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance](https://mlsanity.com/paper/2312.03018/) (2023)
- [NTIRE 2024 Quality Assessment of AI-Generated Content Challenge](https://mlsanity.com/paper/2404.16687/) (2024)
- [LoViC: Efficient Long Video Generation with Context Compression](https://mlsanity.com/paper/2507.12952/) (2025)
- [Fashion-VDM: Video Diffusion Model for Virtual Try-On](https://mlsanity.com/paper/2411.00225/) (2024)
- [UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control](https://mlsanity.com/paper/2403.02332/) (2024)
- [EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation](https://mlsanity.com/paper/2408.13005/) (2024)
- [WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts](https://mlsanity.com/paper/2601.21282/) (2026)
- [Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation](https://mlsanity.com/paper/2502.00500/) (2025)
- [FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise](https://mlsanity.com/paper/2502.03496/) (2025)

## Abstract (arXiv)

Inspired by the remarkable success of Latent Diffusion Models (LDMs) for image synthesis, we study LDM for text-to-video generation, which is a formidable challenge due to the computational and memory constraints during both model training and inference. A single LDM is usually only capable of generating a very limited number of video frames. Some existing works focus on separate prediction models for generating more video frames, which suffer from additional training cost and frame-level jittering, however. In this paper, we propose a framework called "Reuse and Diffuse" dubbed textitVidRD to produce more frames following the frames already generated by an LDM. Conditioned on an initial video clip with a small number of frames, additional frames are iteratively generated by reusing the original latent features and following the previous diffusion process. Besides, for the autoencoder used for translation between pixel space and latent space, we inject temporal layers into its decoder and fine-tune these layers for higher temporal consistency. We also propose a set of strategies for composing video-text data that involve diverse content from multiple existing datasets including video datasets for action recognition and image-text datasets. Extensive experiments show that our method achieves good results in both quantitative and qualitative evaluations. Our project page is available hrefhttps://anonymous0x233.github.io/ReuseAndDiffuse/here.