# RefCaptioner: Multi-Reference Image-Grounded Video Captioning

arXiv: https://arxiv.org/abs/2607.28509 · HTML: https://mlsanity.com/paper/2607.28509/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 42 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Builds on (most cited first, 30 of 38)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025)
- [Kwai Keye-VL 1.5 Technical Report](https://mlsanity.com/paper/2509.01563/) (2025)
- [Multi-subject Open-set Personalization in Video Generation](https://mlsanity.com/paper/2501.06187/) (2025)
- [ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts](https://mlsanity.com/paper/2507.20939/) (2025)
- [LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence](https://mlsanity.com/paper/2605.25979/) (2026)
- [Debiasing Multimodal Large Language Models via Penalization of Language Priors](https://mlsanity.com/paper/2403.05262/) (2024)
- [Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance](https://mlsanity.com/paper/2603.02175/) (2026)
- [Mavors: Multi-granularity Video Representation for Multimodal Large Language Model](https://mlsanity.com/paper/2504.10068/) (2025)
- [Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts](https://mlsanity.com/paper/2502.07802/) (2025)
- [VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation](https://mlsanity.com/paper/2502.12782/) (2025)
- [MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement](https://mlsanity.com/paper/2505.23742/) (2025)
- [MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios](https://mlsanity.com/paper/2505.21333/) (2025)
- [Progress-Aware Video Frame Captioning](https://mlsanity.com/paper/2412.02071/) (2024)
- [LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Models](https://mlsanity.com/paper/2502.15393/) (2025)
- [VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation](https://mlsanity.com/paper/2505.23484/) (2025)

## Abstract (arXiv)

Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing 20,000 videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.