# ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning

arXiv: https://arxiv.org/abs/2601.09851 · HTML: https://mlsanity.com/paper/2601.09851/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Builds on (most cited first, 19 of 19)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [T*: Re-thinking Temporal Search for Long-Form Video Understanding](https://mlsanity.com/paper/2504.02259/) (2025)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [KeyVideoLLM: Towards Large-scale Video Keyframe Selection](https://mlsanity.com/paper/2407.03104/) (2024)
- [MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMs](https://mlsanity.com/paper/2501.02885/) (2025)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [Deep Learning for Video-Text Retrieval: a Review](https://mlsanity.com/paper/2302.12552/) (2023)
- [CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features](https://mlsanity.com/paper/2410.07610/) (2024)
- [Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks](https://mlsanity.com/paper/2305.15044/) (2023)
- [Information-Theoretic Distillation for Reference-less Summarization](https://mlsanity.com/paper/2403.13780/) (2024)
- [Any2Any: Incomplete Multimodal Retrieval with Conformal Prediction](https://mlsanity.com/paper/2411.10513/) (2024)
- [VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR](https://mlsanity.com/paper/2505.17423/) (2025)
- [A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video](https://mlsanity.com/paper/2312.01575/) (2023)

## Built on by (most cited first, 2 of 2)

- [CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning](https://mlsanity.com/paper/2605.26967/) (2026)
- [VEGAS: Human-Aligned Video Caption Evaluation via Gaze](https://mlsanity.com/paper/2607.08489/) (2026)

## Abstract (arXiv)

Multimodal video captioning condenses dense footage into a structured format of keyframes and natural language. By creating a cohesive multimodal summary, this approach anchors generative AI in rich semantic evidence and serves as a lightweight proxy for high-efficiency retrieval. However, traditional metrics like BLEU or ROUGE fail to quantify information coverage across disparate modalities, such as comparing a paragraph of text to a sequence of keyframes. To address this, we propose the Video Summary Information Loss (ViSIL) score, an information-theoretic framework that quantifies the video information not captured by a summary via vision-language model (VLM) inference. By measuring the information loss, ViSIL is a unified metric that enables direct comparison across multimodal summary formats despite their structural discrepancies. Our results demonstrate that ViSIL scores show a statistically significant correlation with both human and VLM performance on Video Question Answering (VQA) tasks. ViSIL also enables summary selection to optimize the trade-off between information loss and processing speed, establishing a Pareto-optimal frontier that outperforms text summaries by 7% in VQA accuracy without increasing processing load.