# BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues

arXiv: https://arxiv.org/abs/2407.20341 · HTML: https://mlsanity.com/paper/2407.20341/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 66 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)

## Built on by (most cited first, 13 of 13)

- [Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives](https://mlsanity.com/paper/2503.14604/) (2025)
- [Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training](https://mlsanity.com/paper/2410.07336/) (2024)
- [SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning](https://mlsanity.com/paper/2508.06125/) (2025)
- [EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations](https://mlsanity.com/paper/2506.24016/) (2025)
- [Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization](https://mlsanity.com/paper/2408.14547/) (2024)
- [Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?](https://mlsanity.com/paper/2502.06600/) (2025)
- [VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions](https://mlsanity.com/paper/2509.25818/) (2025)
- [LLM-Free Image Captioning Evaluation in Reference-Flexible Settings](https://mlsanity.com/paper/2512.21582/) (2025)
- [Do Image-Text Metrics Respect Semantic Invariances?](https://mlsanity.com/paper/2605.24702/) (2026)
- [Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2606.29997/) (2026)
- [Fluent and Accurate Image Captioning with a Self-Trained Reward Model](https://mlsanity.com/paper/2408.16827/) (2024)
- [ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs](https://mlsanity.com/paper/2604.03765/) (2026)
- [MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation](https://mlsanity.com/paper/2605.06080/) (2026)

## Abstract (arXiv)

Effectively aligning with human judgment when evaluating machine-generated image captions represents a complex yet intriguing challenge. Existing evaluation metrics like CIDEr or CLIP-Score fall short in this regard as they do not take into account the corresponding image or lack the capability of encoding fine-grained details and penalizing hallucinations. To overcome these issues, in this paper, we propose BRIDGE, a new learnable and reference-free image captioning metric that employs a novel module to map visual features into dense vectors and integrates them into multi-modal pseudo-captions which are built during the evaluation process. This approach results in a multimodal metric that properly incorporates information from the input image without relying on reference captions, bridging the gap between human judgment and machine-generated image captions. Experiments spanning several datasets demonstrate that our proposal achieves state-of-the-art results compared to existing reference-free evaluation scores. Our source code and trained models are publicly available at: https://github.com/aimagelab/bridge-score.