# Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM

arXiv: https://arxiv.org/abs/2412.15156 · HTML: https://mlsanity.com/paper/2412.15156/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 50 of its 58 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)

## Built on by (most cited first, 25 of 25)

- [Seedance 1.0: Exploring the Boundaries of Video Generation Models](https://mlsanity.com/paper/2506.09113/) (2025)
- [Goku: Flow Based Video Generative Foundation Models](https://mlsanity.com/paper/2502.04896/) (2025)
- [FlashVideo: Flowing Fidelity to Detail for Efficient High-Resolution Video Generation](https://mlsanity.com/paper/2502.05179/) (2025)
- [VISTA: A Test-Time Self-Improving Video Generation Agent](https://mlsanity.com/paper/2510.15831/) (2025)
- [VPO: Aligning Text-to-Video Generation Models with Prompt Optimization](https://mlsanity.com/paper/2503.20491/) (2025)
- [VQQA: An Agentic Approach for Video Evaluation and Quality Improvement](https://mlsanity.com/paper/2603.12310/) (2026)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [Diverse Video Generation with Determinantal Point Process-Guided Policy Optimization](https://mlsanity.com/paper/2511.20647/) (2025)
- [PhyPrompt: RL-based Prompt Refinement for Physically Plausible Text-to-Video Generation](https://mlsanity.com/paper/2603.03505/) (2026)
- [BrandFusion: A Multi-Agent Framework for Seamless Brand Integration in Text-to-Video Generation](https://mlsanity.com/paper/2603.02816/) (2026)
- [CAPE-T2V: Captioner-Anchored Prompt Enhancement toward Two-Sided Conditioning Alignment in Text-to-Video Generation](https://mlsanity.com/paper/2608.03046/) (2026)
- [LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models](https://mlsanity.com/paper/2512.13290/) (2025)
- [When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generation](https://mlsanity.com/paper/2605.16716/) (2026)
- [Bridging Creative Intent and Visual Quality: Creator-Driven Recurrent Video Generation with Agentic Feedback Loops](https://mlsanity.com/paper/2606.18591/) (2026)
- [Flux-OPD: On-Policy Distillation with Evolving Contexts](https://mlsanity.com/paper/2607.28022/) (2026)
- [Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence](https://mlsanity.com/paper/2608.12290/) (2026)
- [WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation](https://mlsanity.com/paper/2609.30221/) (2026)
- [A Reason-then-Describe Instruction Interpreter for Controllable Video Generation](https://mlsanity.com/paper/2511.20563/) (2025)
- [P-Flow: Prompting Visual Effects Generation](https://mlsanity.com/paper/2603.22091/) (2026)
- [Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey](https://mlsanity.com/paper/2604.11283/) (2026)
- [From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models](https://mlsanity.com/paper/2609.11242/) (2026)
- [Reliable Self-Evolution with Imperfect Proxy Rewards](https://mlsanity.com/paper/2610.02975/) (2026)
- [Towards Unified Evaluation of Prompt Enhancers for Video Generation](https://mlsanity.com/paper/2610.11736/) (2026)
- [TIDE: Trajectory-Informed Skill Distillation for Efficient Text-to-Video Prompt Optimization](https://mlsanity.com/paper/2610.15098/) (2026)
- [FlowCred: Hierarchical Credit Assignment and Functional Coordination for Joint Text-to-Video Post-Training](https://mlsanity.com/paper/2610.18142/) (2026)

## Abstract (arXiv)

Text-to-video models have made remarkable advancements through optimization on high-quality text-video pairs, where the textual prompts play a pivotal role in determining quality of output videos. However, achieving the desired output often entails multiple revisions and iterative inference to refine user-provided prompts. Current automatic methods for refining prompts encounter challenges such as Modality-Inconsistency, Cost-Discrepancy, and Model-Unaware when applied to text-to-video diffusion models. To address these problem, we introduce an LLM-based prompt adaptation framework, termed as Prompt-A-Video, which excels in crafting Video-Centric, Labor-Free and Preference-Aligned prompts tailored to specific video diffusion model. Our approach involves a meticulously crafted two-stage optimization and alignment system. Initially, we conduct a reward-guided prompt evolution pipeline to automatically create optimal prompts pool and leverage them for supervised fine-tuning (SFT) of the LLM. Then multi-dimensional rewards are employed to generate pairwise data for the SFT model, followed by the direct preference optimization (DPO) algorithm to further facilitate preference alignment. Through extensive experimentation and comparative analyses, we validate the effectiveness of Prompt-A-Video across diverse generation models, highlighting its potential to push the boundaries of video generation.