VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Existing Video Detailed Captioning (VDC) methods predominantly rely on costly human annotations or distillation from powerful proprietary models, creating a dependency on external supervision. In this paper, we propose VDC-Agent, an autonomous self-evolving framework that empowers a single Multimodal Large Language Model (MLLM) to generate and refine high-quality captions through principle-guided self-reflection. To overcome the inference latency inherent in iterative refinement, we further propose to internalize this reflective capability into the model. Specifically, we construct VDC-Agent-19K, a preference dataset derived from the agent's self-scored trajectories, and introduce a Curriculum Direct Preference Optimization (DPO) strategy. This strategy leverages the quality gap between generated candidates to progressively align the model from easy to hard samples. Extensive experiments demonstrate that VDC-Agent achieves state-of-the-art performance on VDC and DREAM-1K benchmarks, generating captions with superior detail and faithfulness. Crucially, our internalization strategy retains the inference efficiency of the base model while significantly enhancing its generalization capabilities, as validated by both quantitative metrics and human evaluation.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 51 references that are in the map. First posted on arXiv 24 Nov 2025.
Builds on
45 of 51 references.
Closest ancestors · 2
Same direction, newest first
- 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
- 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agent15 Mar
Other references · 43
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
- 2023VideoChat: Chat-Centric Video Understandingcited by 851
31 more
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
- 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
- 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
- 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
- 2024Understanding the planning of LLM agents: A surveycited by 414
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2020Multi-modal Transformer for Video Retrievalcited by 395
- 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
- 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
- 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
- 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
- 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
- 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
- 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
- 2024Streaming Dense Video Captioningcited by 82
- 2024Vript: A Video Is Worth Thousands of Wordscited by 71
- 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54
- 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
- 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
- 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
- 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
- 2023Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocolscited by 17
- 2024InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Captioncited by 17
- 2025OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Rewardcited by 11
- 2025LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Modelscited by 10
- 2024Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Modelscited by 9
- 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
- 2025Cockatiel: Ensembling Synthetic and Human Preferenced Training for Detailed Video Captioncited by 2
- 2025AVC-DPO: Aligned Video Captioning via Direct Preference Optimizationcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 3.
4 papers.
- 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
- 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
- 2026CaMo: Camera Motion Grounded Evaluation and Training for Vision-Language Modelscited by 0
- 2026CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioningcited by 0
- 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning27 May 2026
- 2026CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning26 May 2026
- 2026CaMo: Camera Motion Grounded Evaluation and Training for Vision-Language Models19 May 2026
- 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoning22 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2026ReflectCAP: Detailed Image Captioning with Reflective Memory
- 2025Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions
- 2026Building a Precise Video Language with Human-AI Oversight
- 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
- 2026Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence
- 2025IF-VidCap: Can Video Caption Models Follow Instructions?
- 2026Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
- 2026AVA-Encoder: Towards Agent-Native Video Representation Learning