mlsanityData to 7 Oct 2026

VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection

arXiv 2511.19436 ↗First posted 24 Nov 2025cs.CV cs.AI cs.LGcited by 4

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Existing Video Detailed Captioning (VDC) methods predominantly rely on costly human annotations or distillation from powerful proprietary models, creating a dependency on external supervision. In this paper, we propose VDC-Agent, an autonomous self-evolving framework that empowers a single Multimodal Large Language Model (MLLM) to generate and refine high-quality captions through principle-guided self-reflection. To overcome the inference latency inherent in iterative refinement, we further propose to internalize this reflective capability into the model. Specifically, we construct VDC-Agent-19K, a preference dataset derived from the agent's self-scored trajectories, and introduce a Curriculum Direct Preference Optimization (DPO) strategy. This strategy leverages the quality gap between generated candidates to progressively align the model from easy to hard samples. Extensive experiments demonstrate that VDC-Agent achieves state-of-the-art performance on VDC and DREAM-1K benchmarks, generating captions with superior detail and faithfulness. Crucially, our internalization strategy retains the inference efficiency of the base model while significantly enhancing its generalization capabilities, as validated by both quantitative metrics and human evaluation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 51 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

45 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
  2. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agent15 Mar

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
31 more
  1. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  2. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  3. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  4. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  5. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  6. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  7. 2024Understanding the planning of LLM agents: A surveycited by 414
  8. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  9. 2020Multi-modal Transformer for Video Retrievalcited by 395
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  11. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  12. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  13. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  14. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  15. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  16. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  17. 2024Streaming Dense Video Captioningcited by 82
  18. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  19. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54
  20. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  21. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  22. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
  23. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  24. 2023Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocolscited by 17
  25. 2024InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Captioncited by 17
  26. 2025OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Rewardcited by 11
  27. 2025LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Modelscited by 10
  28. 2024Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Modelscited by 9
  29. 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
  30. 2025Cockatiel: Ensembling Synthetic and Human Preferenced Training for Detailed Video Captioncited by 2
  31. 2025AVC-DPO: Aligned Video Captioning via Direct Preference Optimizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReflectCAP: Detailed Image Captioning with Reflective Memory
  2. 2025Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions
  3. 2026Building a Precise Video Language with Human-AI Oversight
  4. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
  5. 2026Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence
  6. 2025IF-VidCap: Can Video Caption Models Follow Instructions?
  7. 2026Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
  8. 2026AVA-Encoder: Towards Agent-Native Video Representation Learning