# Clue: Cross-modal Coherence Modeling for Caption Generation

arXiv: https://arxiv.org/abs/2005.00908 · HTML: https://mlsanity.com/paper/2005.00908/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 24 of 24)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [MemeCap: A Dataset for Captioning and Interpreting Memes](https://mlsanity.com/paper/2305.13703/) (2023)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Underspecification in Scene Description-to-Depiction Tasks](https://mlsanity.com/paper/2210.05815/) (2022)
- [Concadia: Towards Image-Based Text Generation with a Purpose](https://mlsanity.com/paper/2104.08376/) (2021)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [COSMic: A Coherence-Aware Generation Metric for Image Descriptions](https://mlsanity.com/paper/2109.05281/) (2021)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [Coherence-Driven Multimodal Safety Dialogue with Active Learning for Embodied Agents](https://mlsanity.com/paper/2410.14141/) (2024)
- [Cross-Modal Coherence for Text-to-Image Retrieval](https://mlsanity.com/paper/2109.11047/) (2021)
- [CLID: Controlled-Length Image Descriptions with Limited Data](https://mlsanity.com/paper/2211.14835/) (2022)
- ["We Demand Justice!": Towards Social Context Grounding of Political Texts](https://mlsanity.com/paper/2311.09106/) (2023)
- [CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?](https://mlsanity.com/paper/2502.11300/) (2025)
- [RONA: Pragmatically Diverse Image Captioning with Coherence Relations](https://mlsanity.com/paper/2503.10997/) (2025)
- [IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection](https://mlsanity.com/paper/2505.16258/) (2025)
- [Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning](https://mlsanity.com/paper/2302.02124/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Multimodal Coherent Explanation Generation of Robot Failures](https://mlsanity.com/paper/2410.00659/) (2024)
- [Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation](https://mlsanity.com/paper/2510.09390/) (2025)
- [AnANet: Modeling Association and Alignment for Cross-modal Correlation Classification](https://mlsanity.com/paper/2109.00693/) (2021)
- [Zero-shot Cross-Linguistic Learning of Event Semantics](https://mlsanity.com/paper/2207.02356/) (2022)
- [ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis](https://mlsanity.com/paper/2404.10141/) (2024)

## Abstract (arXiv)

We use coherence relations inspired by computational models of discourse to study the information needs and goals of image captioning. Using an annotation protocol specifically devised for capturing image--caption coherence relations, we annotate 10,000 instances from publicly-available image--caption pairs. We introduce a new task for learning inferences in imagery and text, coherence relation prediction, and show that these coherence annotations can be exploited to learn relation classifiers as an intermediary step, and also train coherence-aware, controllable image captioning models. The results show a dramatic improvement in the consistency and quality of the generated captions with respect to information needs specified via coherence relations.