# CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness

arXiv: https://arxiv.org/abs/2502.14914 · HTML: https://mlsanity.com/paper/2502.14914/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 48 of its 69 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 48)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)

## Built on by (most cited first, 25 of 25)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation](https://mlsanity.com/paper/2504.14988/) (2025)
- [Wan-Image: Pushing the Boundaries of Generative Visual Intelligence](https://mlsanity.com/paper/2604.19858/) (2026)
- [RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understanding](https://mlsanity.com/paper/2510.27261/) (2025)
- [ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement](https://mlsanity.com/paper/2512.13303/) (2025)
- [VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis](https://mlsanity.com/paper/2509.16538/) (2025)
- [AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation](https://mlsanity.com/paper/2603.28068/) (2026)
- [Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment](https://mlsanity.com/paper/2506.01511/) (2025)
- [ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison](https://mlsanity.com/paper/2605.20278/) (2026)
- [DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions](https://mlsanity.com/paper/2604.05623/) (2026)
- [BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning](https://mlsanity.com/paper/2605.07394/) (2026)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models](https://mlsanity.com/paper/2604.17941/) (2026)
- [Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction](https://mlsanity.com/paper/2605.14569/) (2026)
- [VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning](https://mlsanity.com/paper/2605.28023/) (2026)
- [CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning](https://mlsanity.com/paper/2603.18282/) (2026)
- [Reliability-Prioritized Fine-Grained Generation in Multimodal Large](https://mlsanity.com/paper/2606.29573/) (2026)
- [Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning](https://mlsanity.com/paper/2607.28986/) (2026)
- [CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering](https://mlsanity.com/paper/2608.11074/) (2026)
- [DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos](https://mlsanity.com/paper/2608.29711/) (2026)
- [Human-Centric Image Captioning with Subject-Centered Spatial Understanding](https://mlsanity.com/paper/2609.08300/) (2026)
- [Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understanding](https://mlsanity.com/paper/2609.19767/) (2026)
- [VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration](https://mlsanity.com/paper/2609.21521/) (2026)
- [OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning](https://mlsanity.com/paper/2610.12458/) (2026)
- [PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary](https://mlsanity.com/paper/2610.16066/) (2026)

## Abstract (arXiv)

Visual captioning benchmarks have become outdated with the emergence of modern multimodal large language models (MLLMs), as the brief ground-truth sentences and traditional metrics fail to assess detailed captions effectively. While recent benchmarks attempt to address this by focusing on keyword extraction or object-centric evaluation, they remain limited to vague-view or object-view analyses and incomplete visual element coverage. In this paper, we introduce CAPability, a comprehensive multi-view benchmark for evaluating visual captioning across 12 dimensions spanning six critical views. We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions. CAPability stably assesses both the correctness and thoroughness of captions with textitprecision and textithit metrics. By converting annotations to QA pairs, we further introduce a heuristic metric, textitknow but cannot tell (KbarT), indicating a significant performance gap between QA and caption capabilities. Our work provides a holistic analysis of MLLMs' captioning abilities, as we identify their strengths and weaknesses across various dimensions, guiding future research to enhance specific aspects of their capabilities.