# CaptionQA: Is Your Caption as Useful as the Image Itself?

arXiv: https://arxiv.org/abs/2511.21025 · HTML: https://mlsanity.com/paper/2511.21025/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 30 of 31)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Magma: A Foundation Model for Multimodal AI Agents](https://mlsanity.com/paper/2502.13130/) (2025)
- [NVLM: Open Frontier-Class Multimodal LLMs](https://mlsanity.com/paper/2409.11402/) (2024)
- [Describe Anything: Detailed Localized Image and Video Captioning](https://mlsanity.com/paper/2504.16072/) (2025)
- [Benchmarking and Improving Detail Image Caption](https://mlsanity.com/paper/2405.19092/) (2024)
- [Learning to Evaluate Image Captioning](https://mlsanity.com/paper/1806.06422/) (2018)
- [Re-evaluating Automatic Metrics for Image Captioning](https://mlsanity.com/paper/1612.07600/) (2016)
- [CLAIR: Evaluating Image Captions with Large Language Models](https://mlsanity.com/paper/2310.12971/) (2023)
- [CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era](https://mlsanity.com/paper/2503.12329/) (2025)
- [HallE-Control: Controlling Object Hallucination in Large Multimodal Models](https://mlsanity.com/paper/2310.01779/) (2023)
- [Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning](https://mlsanity.com/paper/2412.08614/) (2024)
- [Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning](https://mlsanity.com/paper/2503.07906/) (2025)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [CoLLM: A Large Language Model for Composed Image Retrieval](https://mlsanity.com/paper/2503.19910/) (2025)

## Built on by (most cited first, 15 of 15)

- [RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning](https://mlsanity.com/paper/2603.09160/) (2026)
- [Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models](https://mlsanity.com/paper/2606.27373/) (2026)
- [OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding](https://mlsanity.com/paper/2602.13758/) (2026)
- [Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation](https://mlsanity.com/paper/2607.23125/) (2026)
- [ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison](https://mlsanity.com/paper/2605.20278/) (2026)
- [BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning](https://mlsanity.com/paper/2605.07394/) (2026)
- [Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding](https://mlsanity.com/paper/2609.11899/) (2026)
- [Momentum-Coupled Rubric Adaptation for Detailed Image Captioning](https://mlsanity.com/paper/2609.36893/) (2026)
- [MotionAtlas: Detailed Region Captioning for Motion-Centric Videos](https://mlsanity.com/paper/2606.29531/) (2026)
- [CAPEval: A Decoupled Caption Evaluation across Understanding and Generation](https://mlsanity.com/paper/2608.02589/) (2026)
- [CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering](https://mlsanity.com/paper/2608.11074/) (2026)
- [AesCanvas: A Large-Scale Dataset and Benchmark for Aesthetic Critique and Contextual Suitability](https://mlsanity.com/paper/2608.26713/) (2026)
- [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](https://mlsanity.com/paper/2609.09973/) (2026)
- [CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video](https://mlsanity.com/paper/2609.17688/) (2026)
- [Learning How to Remember: Memory Modality Selection for GUI Agents](https://mlsanity.com/paper/2610.17928/) (2026)

## Abstract (arXiv)

Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.