# COSMic: A Coherence-Aware Generation Metric for Image Descriptions

arXiv: https://arxiv.org/abs/2109.05281 · HTML: https://mlsanity.com/paper/2109.05281/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering](https://mlsanity.com/paper/1505.05612/) (2015)
- [AI2D-RST: A multimodal corpus of 1000 primary school science diagrams](https://mlsanity.com/paper/1912.03879/) (2019)
- [Learning to Evaluate Image Captioning](https://mlsanity.com/paper/1806.06422/) (2018)
- [Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts](https://mlsanity.com/paper/1904.09073/) (2019)
- [Discern: Discourse-Aware Entailment Reasoning Network for Conversational Machine Reading](https://mlsanity.com/paper/2010.01838/) (2020)
- [Clue: Cross-modal Coherence Modeling for Caption Generation](https://mlsanity.com/paper/2005.00908/) (2020)
- [VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions](https://mlsanity.com/paper/1907.09340/) (2019)
- [Understanding Guided Image Captioning Performance across Domains](https://mlsanity.com/paper/2012.02339/) (2020)
- [Understanding, Categorizing and Predicting Semantic Image-Text Relations](https://mlsanity.com/paper/1906.08595/) (2019)

## Built on by (most cited first, 9 of 9)

- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model](https://mlsanity.com/paper/2406.06004/) (2024)
- [HOTVCOM: Generating Buzzworthy Comments for Videos](https://mlsanity.com/paper/2409.15196/) (2024)
- [HumBEL: A Human-in-the-Loop Approach for Evaluating Demographic Factors of Language Models in Human-Machine Conversations](https://mlsanity.com/paper/2305.14195/) (2023)
- [Cross-Modal Coherence for Text-to-Image Retrieval](https://mlsanity.com/paper/2109.11047/) (2021)
- [LEATHER: A Framework for Learning to Generate Human-like Text in Dialogue](https://mlsanity.com/paper/2210.07777/) (2022)
- [CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?](https://mlsanity.com/paper/2502.11300/) (2025)
- [Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning](https://mlsanity.com/paper/2302.02124/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)

## Abstract (arXiv)

Developers of text generation models rely on automated evaluation metrics as a stand-in for slow and expensive manual evaluations. However, image captioning metrics have struggled to give accurate learned estimates of the semantic and pragmatic success of output text. We address this weakness by introducing the first discourse-aware learned generation metric for evaluating image descriptions. Our approach is inspired by computational theories of discourse for capturing information goals using coherence. We present a dataset of imageunicodex2013description pairs annotated with coherence relations. We then train a coherence-aware metric on a subset of the Conceptual Captions dataset and measure its effectivenessunicodex2014its ability to predict human ratings of output captionsunicodex2014on a test set composed of out-of-domain images. We demonstrate a higher Kendall Correlation Coefficient for our proposed metric with the human judgments for the results of a number of state-of-the-art coherence-aware caption generation models when compared to several other metrics including recently proposed learned metrics such as BLEURT and BERTScore.