# OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Reward

arXiv: https://arxiv.org/abs/2508.18634 · HTML: https://mlsanity.com/paper/2508.18634/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.06958/) (2025)
- [StableVideo: Text-driven Consistency-aware Diffusion Video Editing](https://mlsanity.com/paper/2308.09592/) (2023)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding](https://mlsanity.com/paper/2501.07888/) (2025)
- [Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge](https://mlsanity.com/paper/2501.13468/) (2025)
- [Vript: A Video Is Worth Thousands of Words](https://mlsanity.com/paper/2406.06040/) (2024)
- [VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners](https://mlsanity.com/paper/2212.04979/) (2022)
- [ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling](https://mlsanity.com/paper/2402.06118/) (2024)
- [VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking](https://mlsanity.com/paper/2506.01725/) (2025)
- [Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning](https://mlsanity.com/paper/2503.07906/) (2025)
- [VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation](https://mlsanity.com/paper/2502.12782/) (2025)
- [CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval](https://mlsanity.com/paper/2501.00513/) (2024)

## Built on by (most cited first, 12 of 12)

- [AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration](https://mlsanity.com/paper/2510.10395/) (2025)
- [DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models](https://mlsanity.com/paper/2601.19267/) (2026)
- [TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions](https://mlsanity.com/paper/2602.08711/) (2026)
- [VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection](https://mlsanity.com/paper/2511.19436/) (2025)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [Building a Precise Video Language with Human-AI Oversight](https://mlsanity.com/paper/2604.21718/) (2026)
- [Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning](https://mlsanity.com/paper/2607.01667/) (2026)
- [Claim-Level Rubric Rewards for Video Caption Reinforcement Learning](https://mlsanity.com/paper/2607.05150/) (2026)
- [CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales](https://mlsanity.com/paper/2606.21949/) (2026)
- [VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning](https://mlsanity.com/paper/2605.28023/) (2026)
- [PercepCap: Video Captioner with Structured Spatio-Temporal Perception](https://mlsanity.com/paper/2607.20389/) (2026)
- [PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary](https://mlsanity.com/paper/2610.16066/) (2026)

## Abstract (arXiv)

Video captioning aims to generate comprehensive and coherent descriptions of the video content, contributing to the advancement of both video understanding and generation. However, existing methods often suffer from motion-detail imbalance, as models tend to overemphasize one aspect while neglecting the other. This imbalance results in incomplete captions, which in turn leads to a lack of consistency in video understanding and generation. To address this issue, we propose solutions from two aspects: 1) Data aspect: We constructed the Harmonizing Motion-Detail 270K (HMD-270K) dataset through a two-stage pipeline: Motion-Detail Fusion (MDF) and Fine-Grained Examination (FGE). 2) Optimization aspect: We introduce the Caption Set Equivalence Reward (CSER) based on Group Relative Policy Optimization (GRPO). CSER enhances completeness and accuracy in capturing both motion and details through unit-to-set matching and bidirectional validation. Based on the HMD-270K supervised fine-tuning and GRPO post-training with CSER, we developed OwlCap, a powerful video captioning multi-modal large language model (MLLM) with motion-detail balance. Experimental results demonstrate that OwlCap achieves significant improvements compared to baseline models on two benchmarks: the detail-focused VDC (+4.2 Acc) and the motion-focused DREAM-1K (+4.6 F1). The HMD-270K dataset and OwlCap model will be publicly released to facilitate video captioning research community advancements.