mlsanityData to 14 Oct 2026

Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives

arXiv 2503.14604 ↗First posted 18 Mar 2025cs.CV cs.AI cs.CLcited by 18

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The evaluation of machine-generated image captions is a complex and evolving challenge. With the advent of Multimodal Large Language Models (MLLMs), image captioning has become a core task, increasing the need for robust and reliable evaluation metrics. This survey provides a comprehensive overview of advancements in image captioning evaluation, analyzing the evolution, strengths, and limitations of existing metrics. We assess these metrics across multiple dimensions, including correlation with human judgment, ranking accuracy, and sensitivity to hallucinations. Additionally, we explore the challenges posed by the longer and more detailed captions generated by MLLMs and examine the adaptability of current metrics to these stylistic variations. Our analysis highlights some limitations of standard evaluation approaches and suggests promising directions for future research in image captioning assessment.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 56 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

45 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  2. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model10 Jun
  3. 2024What matters when building vision-language models?3 May

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2023Segment Anythingcited by 8,234
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
30 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  7. 2016Self-critical Sequence Training for Image Captioningcited by 547
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  10. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  11. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  12. 2019Attention on Attention for Image Captioningcited by 289
  13. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  14. 2020X-Linear Attention Networks for Image Captioningcited by 198
  15. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  16. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  17. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  18. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  19. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  20. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  21. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 45
  22. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  23. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  24. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  25. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  26. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  27. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  28. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  29. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  30. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 43
  2. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  3. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 6
  4. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 5
  5. 2025CapGeo-Bench: Decoupling Visual Perception from Reasoning and Evaluating Geometric Understandingcited by 5
  6. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  7. 2025UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understandingcited by 4
  8. 2026TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuningcited by 3
  9. 2025FPBench: A Comprehensive Benchmark of Multimodal Large Language Models for Fingerprint Analysiscited by 1
  10. 2025From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFscited by 1
  11. 2026Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioningcited by 1
  12. 2025Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learningcited by 0
6 more
  1. 2026Visual Model Checking: Graph-Based Inference of Visual Routines for Image Retrievalcited by 0
  2. 2026CIAN: Multi-Stage Framework for Event-Enriched Image Captioning via Retrieval-Augmented Generationcited by 0
  3. 2026Symbal: Detecting Systematic Misalignments in Model-Generated Captionscited by 0
  4. 2026LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learningcited by 0
  5. 2026A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captionscited by 0
  6. 2026VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metriccited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
  2. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
  3. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  4. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  5. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  6. 2024Benchmarking and Improving Detail Image Caption
  7. 2023CLAIR: Evaluating Image Captions with Large Language Models
  8. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage