mlsanityData to 14 Oct 2026

Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation

arXiv 2503.24379 ↗First posted 31 Mar 2025cs.CV cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

To address the bottleneck of accurate user intent interpretation within the current video generation community, we present Any2Caption, a novel framework for controllable video generation under any condition. The key idea is to decouple various condition interpretation steps from the video synthesis step. By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs--text, images, videos, and specialized cues such as region, motion, and camera poses--into dense, structured captions that offer backbone video generators with better guidance. We also introduce Any2CapIns, a large-scale dataset with 337K instances and 407K conditions for any-condition-to-caption instruction tuning. Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models. Project Page: https://sqwu.top/Any2Cap/

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 64 of its 76 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

64 of 76 references.

References · 64

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  10. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  12. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
38 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  2. 2024Depth Anything V2cited by 1,422
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  10. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  11. 2024Qwen2-Audio Technical Reportcited by 560
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  13. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  14. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  15. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  16. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  17. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  18. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  19. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  20. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  21. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  22. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  23. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  24. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  25. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  26. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  27. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  28. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 140
  29. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  30. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  31. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  32. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  33. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  34. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 114
  35. 2022Generating Long Videos of Dynamic Scenescited by 113
  36. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  37. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 89
  38. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 2.

10 papers.

  1. 2025Controllable Video Generation: A Surveycited by 98
  2. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
  3. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  4. 2025MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMscited by 11
  5. 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generationcited by 7
  6. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  7. 2025In-Video Instructions: Visual Signals as Generative Controlcited by 1
  8. 2026Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Disseminationcited by 1
  9. 2025A Reason-then-Describe Instruction Interpreter for Controllable Video Generationcited by 0
  10. 2026Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025IF-VidCap: Can Video Caption Models Follow Instructions?
  2. 2025AnyI2V: Animating Any Conditional Image with Motion Control
  3. 2025UniVideo: Unified Understanding, Generation, and Editing for Videos
  4. 2026Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions
  5. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing
  6. 2026OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning
  7. 2025UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark
  8. 2026Building a Precise Video Language with Human-AI Oversight