mlsanityData to 14 Oct 2026

CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness

arXiv 2502.14914 ↗First posted 19 Feb 2025cs.CV cs.CL cs.LGcited by 25

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual captioning benchmarks have become outdated with the emergence of modern multimodal large language models (MLLMs), as the brief ground-truth sentences and traditional metrics fail to assess detailed captions effectively. While recent benchmarks attempt to address this by focusing on keyword extraction or object-centric evaluation, they remain limited to vague-view or object-view analyses and incomplete visual element coverage. In this paper, we introduce CAPability, a comprehensive multi-view benchmark for evaluating visual captioning across 12 dimensions spanning six critical views. We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions. CAPability stably assesses both the correctness and thoroughness of captions with textitprecision and textithit metrics. By converting annotations to QA pairs, we further introduce a heuristic metric, textitknow but cannot tell (KbarT), indicating a significant performance gap between QA and caption capabilities. Our work provides a holistic analysis of MLLMs' captioning abilities, as we identify their strengths and weaknesses across various dimensions, guiding future research to enhance specific aspects of their capabilities.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 69 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

48 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning11 Dec
  2. 2024NVILA: Efficient Frontier Visual Language Models5 Dec
  3. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark14 Feb

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
33 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  11. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  14. 2023ModelScope Text-to-Video Technical Reportcited by 658
  15. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  16. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  17. 2018nocaps: novel object captioning at scalecited by 444
  18. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  19. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  20. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  21. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  22. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  23. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  24. 2014From Captions to Visual Concepts and Backcited by 109
  25. 2022PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR Systemcited by 106
  26. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  27. 2024Benchmarking and Improving Detail Image Captioncited by 81
  28. 2013Recognizing Image Stylecited by 64
  29. 2023Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrievalcited by 27
  30. 2024CoReS: Orchestrating the Dance of Reasoning and Segmentationcited by 25
  31. 2023Efficient Decision-based Black-box Patch Attacks on Video Recognitioncited by 21
  32. 2022Visual Recognition by Requestcited by 10
  33. 2025VideoPure: Diffusion-based Adversarial Purification for Video Recognitioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 19.

25 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluationcited by 19
  3. 2026Wan-Image: Pushing the Boundaries of Generative Visual Intelligencecited by 16
  4. 2025RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understandingcited by 8
  5. 2025ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinementcited by 5
  6. 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysiscited by 4
  7. 2026AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generationcited by 4
  8. 2025Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignmentcited by 3
  9. 2026ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparisoncited by 3
  10. 2026DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captionscited by 2
  11. 2026BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioningcited by 2
  12. 2026ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Modelscited by 2
13 more
  1. 2026From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Modelscited by 1
  2. 2026Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstructioncited by 1
  3. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  4. 2026CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuningcited by 0
  5. 2026Reliability-Prioritized Fine-Grained Generation in Multimodal Largecited by 0
  6. 2026Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioningcited by 0
  7. 2026CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answeringcited by 0
  8. 2026DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videoscited by 0
  9. 2026Human-Centric Image Captioning with Subject-Centered Spatial Understandingcited by 0
  10. 2026Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understandingcited by 0
  11. 2026VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Durationcited by 0
  12. 2026OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioningcited by 0
  13. 2026PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summarycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  2. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
  3. 2026VIVECaption: A Split Approach to Caption Quality Improvement
  4. 2025VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
  5. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  6. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
  7. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
  8. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction