# Grid Diffusion Models for Text-to-Video Generation

arXiv: https://arxiv.org/abs/2404.00234 · HTML: https://mlsanity.com/paper/2404.00234/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 39 of its 46 references that are in the map.
- First posted on arXiv 30 Mar 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](https://mlsanity.com/paper/2305.10474/) (2023)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)

## Built on by (most cited first, 21 of 21)

- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [Code2Worlds: Empowering Coding LLMs for 4D World Generation](https://mlsanity.com/paper/2602.11757/) (2026)
- [Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation](https://mlsanity.com/paper/2411.16810/) (2024)
- [MedSegFactory: Text-Guided Generation of Medical Image-Mask Pairs](https://mlsanity.com/paper/2504.06897/) (2025)
- [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [SwitchCraft: Training-Free Multi-Event Video Generation with Attention Controls](https://mlsanity.com/paper/2602.23956/) (2026)
- [X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Rendering](https://mlsanity.com/paper/2510.08530/) (2025)
- [Identity Preserving 3D Head Stylization with Multiview Score Distillation](https://mlsanity.com/paper/2411.13536/) (2024)
- [Bridging Text and Video Generation: A Survey](https://mlsanity.com/paper/2510.04999/) (2025)
- [Bring Your Dreams to Life: Continual Text-to-Video Customization](https://mlsanity.com/paper/2512.05802/) (2025)
- [Accelerating Diffusion Sampling via Exploiting Local Transition Coherence](https://mlsanity.com/paper/2503.09675/) (2025)
- [From Prompt to Progression: Taming Video Diffusion Models for Seamless Attribute Transition](https://mlsanity.com/paper/2509.19690/) (2025)
- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)
- [ContrastiveGaussian: High-Fidelity 3D Generation with Contrastive Learning and Gaussian Splatting](https://mlsanity.com/paper/2504.08100/) (2025)
- [Multimodal Cinematic Video Synthesis Using Text-to-Image and Audio Generation Models](https://mlsanity.com/paper/2506.10005/) (2025)
- [StgcDiff: Spatial-Temporal Graph Condition Diffusion for Sign Language Transition Generation](https://mlsanity.com/paper/2506.13156/) (2025)
- [Progressive Image Restoration via Text-Conditioned Video Generation](https://mlsanity.com/paper/2512.02273/) (2025)
- [CultureScore: Evaluating Cultural Faithfulness in Video Generation Models](https://mlsanity.com/paper/2606.07311/) (2026)
- [Variational Test-time Optimization for Diffusion Synchronization](https://mlsanity.com/paper/2606.15614/) (2026)
- [Multi-Grid Post-Training for Long-Form Multi-Shot Video Generation](https://mlsanity.com/paper/2609.06373/) (2026)

## Abstract (arXiv)

Recent advances in the diffusion models have significantly improved text-to-image generation. However, generating videos from text is a more challenging task than generating images from text, due to the much larger dataset and higher computational cost required. Most existing video generation methods use either a 3D U-Net architecture that considers the temporal dimension or autoregressive generation. These methods require large datasets and are limited in terms of computational costs compared to text-to-image generation. To tackle these challenges, we propose a simple but effective novel grid diffusion for text-to-video generation without temporal dimension in architecture and a large text-video paired dataset. We can generate a high-quality video using a fixed amount of GPU memory regardless of the number of frames by representing the video as a grid image. Additionally, since our method reduces the dimensions of the video to the dimensions of the image, various image-based methods can be applied to videos, such as text-guided video manipulation from image manipulation. Our proposed method outperforms the existing methods in both quantitative and qualitative evaluations, demonstrating the suitability of our model for real-world video generation.