# VideoBooth: Diffusion-based Video Generation with Image Prompts

arXiv: https://arxiv.org/abs/2312.00777 · HTML: https://mlsanity.com/paper/2312.00777/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 89 of its 101 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 89)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)

## Built on by (most cited first, 30 of 126)

- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [Phantom: Subject-consistent video generation via cross-modal alignment](https://mlsanity.com/paper/2502.11079/) (2025)
- [ID-Animator: Zero-Shot Identity-Preserving Human Video Generation](https://mlsanity.com/paper/2404.15275/) (2024)
- [Long Context Tuning for Video Generation](https://mlsanity.com/paper/2503.10589/) (2025)
- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [MotionBooth: Motion-Aware Customized Text-to-Video Generation](https://mlsanity.com/paper/2406.17758/) (2024)
- [HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation](https://mlsanity.com/paper/2505.04512/) (2025)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects](https://mlsanity.com/paper/2401.09962/) (2024)
- [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025)
- [Training-free and Adaptive Sparse Attention for Efficient Long Video Generation](https://mlsanity.com/paper/2502.21079/) (2025)
- [ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning](https://mlsanity.com/paper/2501.04698/) (2025)
- [Multi-subject Open-set Personalization in Video Generation](https://mlsanity.com/paper/2501.06187/) (2025)
- [CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities](https://mlsanity.com/paper/2408.13239/) (2024)
- [RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control](https://mlsanity.com/paper/2502.10059/) (2025)
- [HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives](https://mlsanity.com/paper/2510.20822/) (2025)
- [CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility](https://mlsanity.com/paper/2403.12035/) (2024)
- [Still-Moving: Customized Video Generation without Customized Video Data](https://mlsanity.com/paper/2407.08674/) (2024)
- [UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing](https://mlsanity.com/paper/2402.13185/) (2024)
- [DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control](https://mlsanity.com/paper/2410.13830/) (2024)
- [Mind the Time: Temporally-Controlled Multi-Event Video Generation](https://mlsanity.com/paper/2412.05263/) (2024)
- [TrackGo: A Flexible and Efficient Method for Controllable Video Generation](https://mlsanity.com/paper/2408.11475/) (2024)
- [Reangle-A-Video: 4D Video Generation as Video-to-Video Translation](https://mlsanity.com/paper/2503.09151/) (2025)
- [OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory](https://mlsanity.com/paper/2512.07802/) (2025)
- [Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation](https://mlsanity.com/paper/2403.09625/) (2024)
- [SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control](https://mlsanity.com/paper/2403.19438/) (2024)
- [A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training](https://mlsanity.com/paper/2405.17403/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts](https://mlsanity.com/paper/2502.07802/) (2025)
- [PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data](https://mlsanity.com/paper/2502.14397/) (2025)

## Abstract (arXiv)

Text-driven video generation witnesses rapid progress. However, merely using text prompts is not enough to depict the desired subject appearance that accurately aligns with users' intents, especially for customized content creation. In this paper, we study the task of video generation with image prompts, which provide more accurate and direct content control beyond the text prompts. Specifically, we propose a feed-forward framework VideoBooth, with two dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine manner. Coarse visual embeddings from image encoder provide high-level encodings of image prompts, while fine visual embeddings from the proposed attention injection module provide multi-scale and detailed encoding of image prompts. These two complementary embeddings can faithfully capture the desired appearance. 2) In the attention injection module at fine level, multi-scale image prompts are fed into different cross-frame attention layers as additional keys and values. This extra spatial information refines the details in the first frame and then it is propagated to the remaining frames, which maintains temporal consistency. Extensive experiments demonstrate that VideoBooth achieves state-of-the-art performance in generating customized high-quality videos with subjects specified in image prompts. Notably, VideoBooth is a generalizable framework where a single model works for a wide range of image prompts with feed-forward pass.