mlsanityData to 7 Oct 2026

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model

arXiv 2406.06004 ↗First posted 10 Jun 2024cs.CV cs.AI cs.CLcited by 44

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Most existing image captioning evaluation metrics focus on assigning a single numerical score to a caption by comparing it with reference captions. However, these methods do not provide an explanation for the assigned score. Moreover, reference captions are expensive to acquire. In this paper, we propose FLEUR, an explainable reference-free metric to introduce explainability into image captioning evaluation metrics. By leveraging a large multimodal model, FLEUR can evaluate the caption against the image without the need for reference captions, and provide the explanation for the assigned score. We introduce score smoothing to align as closely as possible with human judgment and to be robust to user-defined grading criteria. FLEUR achieves high correlations with human judgment across various image captioning evaluation benchmarks and reaches state-of-the-art results on Flickr8k-CF, COMPOSITE, and Pascal-50S within the domain of reference-free evaluation metrics. Our source code and results are publicly available at: https://github.com/Yebin46/FLEUR.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 34 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

21 of 34 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  12. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
9 more
  1. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  2. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  3. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  4. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  5. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  6. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  7. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  8. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  9. 2021COSMic: A Coherence-Aware Generation Metric for Image Descriptionscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 15.

44 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 40
  3. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 23
  4. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  5. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 17
  6. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 17
  7. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  8. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  9. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 13
  10. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  11. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  12. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
32 more
  1. 2024I0T: Embedding Standardization Method Towards Zero Modality Gapcited by 7
  2. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  3. 2025Quantitative LLM Judgescited by 6
  4. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 6
  5. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  6. 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gapcited by 6
  7. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  8. 2024Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologiescited by 4
  9. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translationcited by 3
  10. 2024Diagram-Driven Course Questions Generationcited by 2
  11. 2024Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Modelscited by 2
  12. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  13. 2025DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioningcited by 2
  14. 2025LLM-Free Image Captioning Evaluation in Reference-Flexible Settingscited by 2
  15. 2025DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinementcited by 1
  16. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  17. 2025Physics-Based Benchmarking Metrics for Multimodal Synthetic Imagescited by 1
  18. 2026Unsafe2Safe: Controllable Image Anonymization for Downstream Utilitycited by 1
  19. 2026MLLM-as-a-Judge Exhibits Model Preference Biascited by 1
  20. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 1
  21. 2025Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulationcited by 0
  22. 2025Generating Accurate and Detailed Captions for High-Resolution Imagescited by 0
  23. 2026Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioningcited by 0
  24. 2026CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluationcited by 0
  25. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMscited by 0
  26. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  27. 2026Evaluating Remote Sensing Image Captions Beyond Metric Biasescited by 0
  28. 2026MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluationcited by 0
  29. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Modelcited by 0
  30. 2026Symbal: Detecting Systematic Misalignments in Model-Generated Captionscited by 0
  31. 2026A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captionscited by 0
  32. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluation
  2. 2024Benchmarking and Improving Detail Image Caption
  3. 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis
  4. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  5. 2023CLAIR: Evaluating Image Captions with Large Language Models
  6. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?
  7. 2024G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o
  8. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions