A Reason-then-Describe Instruction Interpreter for Controllable Video Generation
On the map: Diffusion models → Video generation
Abstract (arXiv)
Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 56 references that are in the map. First posted on arXiv 25 Nov 2025.
Builds on
51 of 56 references.
Closest ancestors · 12
Same direction, newest first
- 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM19 Dec
- 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Model21 Oct
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
- 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model15 Apr
- 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
- 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
- 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects18 Jan
- 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models27 Dec
- 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models7 Dec
- 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motion7 Dec
- 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
- 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
Other references · 39
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
- 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
27 more
- 2025Qwen2.5-Omni Technical Reportcited by 704
- 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
- 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
- 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
- 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
- 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
- 2022Optimizing Prompts for Text-to-Image Generationcited by 204
- 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
- 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
- 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
- 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
- 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
- 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
- 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 102
- 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
- 2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
- 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
- 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
- 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 38
- 2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimizationcited by 27
- 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
- 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
- 2024InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Captioncited by 17
- 2025SketchVideo: Sketch-based Video Generation and Editingcited by 15
- 2024PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Controlcited by 10
- 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generation
- 2024Video Diffusion Transformers are In-Context Learners
- 2026From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models
- 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
- 2026KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing
- 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
- 2025Video-As-Prompt: Unified Semantic Control for Video Generation
- 2026InstructionCrafter: Generating Consistent and High-Fidelity Visual Instructions