mlsanityData to 14 Oct 2026

CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era

arXiv 2503.12329 ↗First posted 16 Mar 2025cs.CV cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just 4 per test. Data and resources will be open-sourced at https://caparena.github.io.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

37 of 53 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
25 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  4. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  10. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  11. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  12. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  13. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  14. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  15. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  16. 2024Benchmarking and Improving Detail Image Captioncited by 81
  17. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
  18. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  19. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
  20. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  21. 2021Transparent Human Evaluation for Image Captioningcited by 40
  22. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 31
  23. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  24. 2023Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Modelcited by 10
  25. 2023ADS-Cap: A Framework for Accurate and Diverse Stylized Captioning with Unpaired Stylistic Corporacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 25.

41 papers.

Sort:

By area:

  1. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thoughtcited by 21
  2. 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Modelcited by 18
  3. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  4. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 15
  5. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  6. 2025TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoningcited by 12
  7. 2025From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Modelcited by 9
  8. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  9. 2026CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioningcited by 8
  10. 2025SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoningcited by 7
  11. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 7
  12. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
29 more
  1. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
  2. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Modelscited by 5
  3. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  4. 2026Test-Time Attention Purification for Backdoored Large Vision Language Modelscited by 5
  5. 2026Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillationcited by 4
  6. 2026How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision Peoplecited by 3
  7. 2025GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Modelscited by 2
  8. 2026BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioningcited by 2
  9. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 2
  10. 2025DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinementcited by 1
  11. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  12. 2026ReflectCAP: Detailed Image Captioning with Reflective Memorycited by 1
  13. 2026Robust Onion: Peeling Open Vocab Object Detectors Under Noisecited by 1
  14. 2025Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Frameworkcited by 0
  15. 2025Skin-SOAP: A Weakly Supervised Framework for Generating Structured SOAP Notescited by 0
  16. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMscited by 0
  17. 2026GaLa: Hypergraph-Guided Visual Language Models for Procedural Planningcited by 0
  18. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  19. 2026Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspectivecited by 0
  20. 2026MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluationcited by 0
  21. 2026Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Modelscited by 0
  22. 2026Reliability-Prioritized Fine-Grained Generation in Multimodal Largecited by 0
  23. 2026A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captionscited by 0
  24. 2026CAPEval: A Decoupled Caption Evaluation across Understanding and Generationcited by 0
  25. 2026CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answeringcited by 0
  26. 2026Latent-Aligned Reasoning for Multimodal Recommendationcited by 0
  27. 2026InterHier: Learning Interconnected Hierarchical Semantics for Open-Vocabulary Object Detectioncited by 0
  28. 2026Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoderscited by 0
  29. 2026Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  2. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  3. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
  4. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
  5. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
  6. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
  7. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  8. 2026VIVECaption: A Split Approach to Caption Quality Improvement