# CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era

arXiv: https://arxiv.org/abs/2503.12329 · HTML: https://mlsanity.com/paper/2503.12329/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://mlsanity.com/paper/2412.19723/) (2024)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [Benchmarking and Improving Detail Image Caption](https://mlsanity.com/paper/2405.19092/) (2024)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)
- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)

## Built on by (most cited first, 30 of 41)

- [When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought](https://mlsanity.com/paper/2511.02779/) (2025)
- [Vision-aligned Latent Reasoning for Multi-modal Large Language Model](https://mlsanity.com/paper/2602.04476/) (2026)
- [OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows](https://mlsanity.com/paper/2510.03506/) (2025)
- [CaptionQA: Is Your Caption as Useful as the Image Itself?](https://mlsanity.com/paper/2511.21025/) (2025)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoning](https://mlsanity.com/paper/2510.03519/) (2025)
- [From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model](https://mlsanity.com/paper/2510.19871/) (2025)
- [RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning](https://mlsanity.com/paper/2603.09160/) (2026)
- [CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning](https://mlsanity.com/paper/2602.21655/) (2026)
- [SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning](https://mlsanity.com/paper/2510.16416/) (2025)
- [Direct Visual Grounding by Directing Attention of Visual Tokens](https://mlsanity.com/paper/2511.12738/) (2025)
- [SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More](https://mlsanity.com/paper/2601.05688/) (2026)
- [Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.03906/) (2025)
- [HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models](https://mlsanity.com/paper/2511.12693/) (2025)
- [Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2512.15885/) (2025)
- [Test-Time Attention Purification for Backdoored Large Vision Language Models](https://mlsanity.com/paper/2603.12989/) (2026)
- [Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation](https://mlsanity.com/paper/2607.23125/) (2026)
- [How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People](https://mlsanity.com/paper/2602.13469/) (2026)
- [GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models](https://mlsanity.com/paper/2512.15560/) (2025)
- [BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning](https://mlsanity.com/paper/2605.07394/) (2026)
- [Momentum-Coupled Rubric Adaptation for Detailed Image Captioning](https://mlsanity.com/paper/2609.36893/) (2026)
- [DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement](https://mlsanity.com/paper/2506.15583/) (2025)
- [PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions](https://mlsanity.com/paper/2510.19060/) (2025)
- [ReflectCAP: Detailed Image Captioning with Reflective Memory](https://mlsanity.com/paper/2604.12357/) (2026)
- [Robust Onion: Peeling Open Vocab Object Detectors Under Noise](https://mlsanity.com/paper/2606.26734/) (2026)
- [Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework](https://mlsanity.com/paper/2506.10328/) (2025)
- [Skin-SOAP: A Weakly Supervised Framework for Generating Structured SOAP Notes](https://mlsanity.com/paper/2508.05019/) (2025)
- [ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs](https://mlsanity.com/paper/2604.03765/) (2026)
- [GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning](https://mlsanity.com/paper/2604.17241/) (2026)
- [Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts](https://mlsanity.com/paper/2604.18091/) (2026)

## Abstract (arXiv)

Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just 4 per test. Data and resources will be open-sourced at https://caparena.github.io.