# CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

arXiv: https://arxiv.org/abs/1909.05506 · HTML: https://mlsanity.com/paper/1909.05506/

## Key facts

- Cited by 110 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 30 of 110)

- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [VinVL: Revisiting Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2101.00529/) (2021)
- [Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification](https://mlsanity.com/paper/2107.12666/) (2021)
- [Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models](https://mlsanity.com/paper/2307.14061/) (2023)
- [Vision-Language Models in Remote Sensing: Current Progress and Future Trends](https://mlsanity.com/paper/2305.05726/) (2023)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval](https://mlsanity.com/paper/2204.09868/) (2022)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [Image-text Retrieval: A Survey on Recent Research and Development](https://mlsanity.com/paper/2203.14713/) (2022)
- [Discover and Mitigate Unknown Biases with Debiasing Alternate Networks](https://mlsanity.com/paper/2207.10077/) (2022)
- [LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval](https://mlsanity.com/paper/2103.08784/) (2021)
- [Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching](https://mlsanity.com/paper/2311.12751/) (2023)
- [Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information](https://mlsanity.com/paper/2204.09860/) (2022)
- [Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval](https://mlsanity.com/paper/2304.03391/) (2023)
- [Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory](https://mlsanity.com/paper/2403.12445/) (2024)
- [Improving Cross-Modal Retrieval with Set of Diverse Embeddings](https://mlsanity.com/paper/2211.16761/) (2022)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Exploring Visual Interpretability for Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2209.07046/) (2022)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval](https://mlsanity.com/paper/2308.12509/) (2023)
- [Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics](https://mlsanity.com/paper/2202.00259/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Cross-modal Active Complementary Learning with Self-refining Correspondence](https://mlsanity.com/paper/2310.17468/) (2023)
- [Learning Relation Alignment for Calibrated Cross-modal Retrieval](https://mlsanity.com/paper/2105.13868/) (2021)
- [Cross-Modal Retrieval Augmentation for Multi-Modal Classification](https://mlsanity.com/paper/2104.08108/) (2021)
- [Partially Does It: Towards Scene-Level FG-SBIR with Partial Input](https://mlsanity.com/paper/2203.14804/) (2022)
- [Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrieval](https://mlsanity.com/paper/2203.03853/) (2022)

## Abstract (arXiv)

Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In this paper, we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively controls the information flow for message passing across modalities. Our approach not only takes comprehensive and fine-grained cross-modal interactions into account, but also properly handles negative pairs and irrelevant information with an adaptive gating scheme. Moreover, instead of conventional joint embedding approaches for text-image matching, we infer the matching score based on the fused features, and propose a hardest negative binary cross-entropy loss for training. Results on COCO and Flickr30k significantly surpass state-of-the-art methods, demonstrating the effectiveness of our approach.