mlsanityData to 7 Oct 2026

DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning

arXiv 2409.19255 ↗First posted 28 Sep 2024cs.CV cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In this work, we address the challenge of developing automatic evaluation metrics for image captioning, with a particular focus on robustness against hallucinations. Existing metrics are often inadequate for handling hallucinations, primarily due to their limited ability to compare candidate captions with multifaceted reference captions. To address this shortcoming, we propose DENEB, a novel supervised automatic evaluation metric specifically robust against hallucinations. DENEB incorporates the Sim-Vec Transformer, a mechanism that processes multiple references simultaneously, thereby efficiently capturing the similarity between an image, a candidate caption, and reference captions. To train DENEB, we construct the diverse and balanced Nebula dataset comprising 32,978 images, paired with human judgments provided by 805 annotators. We demonstrated that DENEB achieves state-of-the-art performance among existing LLM-free metrics on the FOIL, Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and PASCAL-50S datasets, validating its effectiveness and robustness against hallucinations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

53 of 67 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  13. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  14. 2018nocaps: novel object captioning at scalecited by 444
  15. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  16. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  17. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  18. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  19. 2020Captioning Images Taken by People Who Are Blindcited by 108
  20. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  21. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  22. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  23. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  24. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  25. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  26. 2018Learning to Evaluate Image Captioningcited by 53
  27. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  28. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 44
  29. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  30. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  31. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  32. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  33. 2019A Survey on Biomedical Image Captioningcited by 30
  34. 2021CIDEr-R: Robust Consensus-based Image Description Evaluationcited by 22
  35. 2021SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysiscited by 18
  36. 2020Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challengecited by 17
  37. 2022SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakescited by 9
  38. 2021Multi-Modal Image Captioning for the Visually Impairedcited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 17
  2. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 6
  3. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  4. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  5. 2025DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioningcited by 2
  6. 2025LLM-Free Image Captioning Evaluation in Reference-Flexible Settingscited by 2
  7. 2026MLLM-as-a-Judge Exhibits Model Preference Biascited by 1
  8. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 1
  9. 2025Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulationcited by 0
  10. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Modelcited by 0
  11. 2026Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluationcited by 0