# CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning

arXiv: https://arxiv.org/abs/2605.26967 · HTML: https://mlsanity.com/paper/2605.26967/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 30 of 32)

- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](https://mlsanity.com/paper/2407.06358/) (2024)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding](https://mlsanity.com/paper/2501.07888/) (2025)
- [TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models](https://mlsanity.com/paper/2410.23266/) (2024)
- [Vript: A Video Is Worth Thousands of Words](https://mlsanity.com/paper/2406.06040/) (2024)
- [Streamlined Dense Video Captioning](https://mlsanity.com/paper/1904.03870/) (2019)
- [STORM: Token-Efficient Long Video Understanding for Multimodal LLMs](https://mlsanity.com/paper/2503.04130/) (2025)
- [End-to-end Dense Video Captioning as Sequence Generation](https://mlsanity.com/paper/2204.08121/) (2022)
- [LVD-2M: A Long-take Video Dataset with Temporally Dense Captions](https://mlsanity.com/paper/2410.10816/) (2024)
- [MR. Video: "MapReduce" is the Principle for Long Video Understanding](https://mlsanity.com/paper/2504.16082/) (2025)
- [Models See Hallucinations: Evaluating the Factuality in Video Captioning](https://mlsanity.com/paper/2303.02961/) (2023)
- [VidHal: Benchmarking Temporal Hallucinations in Vision LLMs](https://mlsanity.com/paper/2411.16771/) (2024)
- [SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis](https://mlsanity.com/paper/2411.16173/) (2024)
- [TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos](https://mlsanity.com/paper/2505.20124/) (2025)
- [Towards Comprehensive Interactive Change Understanding in Remote Sensing: A Large-scale Dataset and Dual-granularity Enhanced VLM](https://mlsanity.com/paper/2509.23105/) (2025)
- [ARC-Chapter: Structuring Hour-Long Videos into Navigable Chapters and Hierarchical Summaries](https://mlsanity.com/paper/2511.14349/) (2025)
- [ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Access](https://mlsanity.com/paper/2511.18382/) (2025)
- [VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection](https://mlsanity.com/paper/2511.19436/) (2025)
- [VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding](https://mlsanity.com/paper/2601.07290/) (2026)
- [OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models](https://mlsanity.com/paper/2507.19262/) (2025)
- [Hallucination Localization in Video Captioning](https://mlsanity.com/paper/2510.25225/) (2025)
- [ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning](https://mlsanity.com/paper/2601.09851/) (2026)
- [HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing](https://mlsanity.com/paper/2412.05685/) (2024)

## Abstract (arXiv)

Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.