mlsanityData to 14 Oct 2026

VC4VG: Optimizing Video Captions for Text-to-Video Generation

arXiv 2510.24134 ↗First posted 28 Oct 2025cs.CV cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent advances in text-to-video (T2V) generation highlight the critical role of high-quality video-text pairs in training models capable of producing coherent and instruction-aligned videos. However, strategies for optimizing video captions specifically for T2V training remain underexplored. In this paper, we introduce VC4VG (Video Captioning for Video Generation), a comprehensive caption optimization framework tailored to the needs of T2V models. We begin by analyzing caption content from a T2V perspective, decomposing the essential elements required for video reconstruction into multiple dimensions, and proposing a principled caption design methodology. To support evaluation, we construct VC4VG-Bench, a new benchmark featuring fine-grained, multi-dimensional, and necessity-graded metrics aligned with T2V-specific requirements. Extensive T2V fine-tuning experiments demonstrate a strong correlation between improved caption quality and video generation performance, validating the effectiveness of our approach. We release all benchmark tools and code at https://github.com/alimama-creative/VC4VG to support further research.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

29 of 46 references.

References · 29

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  12. 2017Localizing Moments in Video with Natural Languagecited by 663
17 more
  1. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  3. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  4. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  5. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  7. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  8. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  9. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  10. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  11. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  12. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  13. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 45
  14. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 23
  15. 2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 19
  16. 2024InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Captioncited by 17
  17. 2024Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrievalcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 13
  2. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 8
  3. 2026Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioningcited by 3
  4. 2026CAPE-T2V: Captioner-Anchored Prompt Enhancement toward Two-Sided Conditioning Alignment in Text-to-Video Generationcited by 3
  5. 2026CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scalescited by 2
  6. 2026P-Flow: Prompting Visual Effects Generationcited by 0
  7. 2026Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generationcited by 0
  8. 2026PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summarycited by 0