# EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations

arXiv: https://arxiv.org/abs/2506.24016 · HTML: https://mlsanity.com/paper/2506.24016/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [TIGEr: Text-to-Image Grounding for Image Caption Evaluation](https://mlsanity.com/paper/1909.02050/) (2019)
- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model](https://mlsanity.com/paper/2406.06004/) (2024)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [LocCa: Visual Pretraining with Location-aware Captioners](https://mlsanity.com/paper/2403.19596/) (2024)
- [From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge](https://mlsanity.com/paper/1511.03292/) (2015)
- [InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation](https://mlsanity.com/paper/2305.06002/) (2023)
- [BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues](https://mlsanity.com/paper/2407.20341/) (2024)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)

## Built on by (most cited first, 7 of 7)

- [RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning](https://mlsanity.com/paper/2603.09160/) (2026)
- [LLM-Free Image Captioning Evaluation in Reference-Flexible Settings](https://mlsanity.com/paper/2512.21582/) (2025)
- [MLLM-as-a-Judge Exhibits Model Preference Bias](https://mlsanity.com/paper/2604.11589/) (2026)
- [Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2606.29997/) (2026)
- [MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation](https://mlsanity.com/paper/2605.06080/) (2026)
- [BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model](https://mlsanity.com/paper/2605.21728/) (2026)
- [VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric](https://mlsanity.com/paper/2610.16441/) (2026)

## Abstract (arXiv)

Recent advances in large language models and vision-language models have led to growing interest in explainable evaluation metrics for image captioning. However, these metrics generate explanations without standardized criteria, and the overall quality of the generated explanations remains unverified. In this paper, we propose EXPERT, a reference-free evaluation metric that provides structured explanations based on three fundamental criteria: fluency, relevance, and descriptiveness. By constructing large-scale datasets of high-quality structured explanations, we develop a two-stage evaluation template to effectively supervise a vision-language model for both scoring and explanation generation. EXPERT achieves state-of-the-art results on benchmark datasets while providing significantly higher-quality explanations than existing metrics, as validated through comprehensive human evaluation. Our code and datasets are available at https://github.com/hjkim811/EXPERT.