mlsanityData to 7 Oct 2026

A Reason-then-Describe Instruction Interpreter for Controllable Video Generation

arXiv 2511.20563 ↗First posted 25 Nov 2025cs.CVcited by 0

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 56 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

51 of 56 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM19 Dec
  2. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Model21 Oct
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  4. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model15 Apr
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
  6. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  7. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects18 Jan
  8. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models27 Dec
  9. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models7 Dec
  10. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motion7 Dec
  11. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  12. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
27 more
  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  3. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  4. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  5. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  6. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  7. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  8. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  9. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  10. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  11. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  12. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  13. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  14. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  15. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 102
  16. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  17. 2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
  18. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  19. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  20. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 38
  21. 2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimizationcited by 27
  22. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
  23. 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
  24. 2024InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Captioncited by 17
  25. 2025SketchVideo: Sketch-based Video Generation and Editingcited by 15
  26. 2024PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Controlcited by 10
  27. 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generation
  2. 2024Video Diffusion Transformers are In-Context Learners
  3. 2026From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models
  4. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
  5. 2026KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing
  6. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
  7. 2025Video-As-Prompt: Unified Semantic Control for Video Generation
  8. 2026InstructionCrafter: Generating Consistent and High-Fidelity Visual Instructions