# Cross-Modal Coherence for Text-to-Image Retrieval

arXiv: https://arxiv.org/abs/2109.11047 · HTML: https://mlsanity.com/paper/2109.11047/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://mlsanity.com/paper/2002.11985/) (2020)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)
- [Engaging Image Captioning Via Personality](https://mlsanity.com/paper/1810.10665/) (2018)
- [Clue: Cross-modal Coherence Modeling for Caption Generation](https://mlsanity.com/paper/2005.00908/) (2020)
- [CookGAN: Meal Image Synthesis from Ingredients](https://mlsanity.com/paper/2002.11493/) (2020)
- [CITE: A Corpus of Image-Text Discourse Relations](https://mlsanity.com/paper/1904.06286/) (2019)
- [COSMic: A Coherence-Aware Generation Metric for Image Descriptions](https://mlsanity.com/paper/2109.05281/) (2021)

## Built on by (most cited first, 5 of 5)

- [IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection](https://mlsanity.com/paper/2505.16258/) (2025)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Data-Efficient Generalization for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2503.05204/) (2025)
- [Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning](https://mlsanity.com/paper/2302.02124/) (2023)
- [A look under the hood of the Interactive Deep Learning Enterprise (No-IDLE)](https://mlsanity.com/paper/2406.19054/) (2024)

## Abstract (arXiv)

Common image-text joint understanding techniques presume that images and the associated text can universally be characterized by a single implicit model. However, co-occurring images and text can be related in qualitatively different ways, and explicitly modeling it could improve the performance of current joint understanding models. In this paper, we train a Cross-Modal Coherence Modelfor text-to-image retrieval task. Our analysis shows that models trained with image--text coherence relations can retrieve images originally paired with target text more often than coherence-agnostic models. We also show via human evaluation that images retrieved by the proposed coherence-aware model are preferred over a coherence-agnostic baseline by a huge margin. Our findings provide insights into the ways that different modalities communicate and the role of coherence relations in capturing commonsense inferences in text and imagery.