# Multimodal Convolutional Neural Networks for Matching Image and Sentence

arXiv: https://arxiv.org/abs/1504.06063 · HTML: https://mlsanity.com/paper/1504.06063/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2015.

## Built on by (most cited first, 25 of 25)

- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [ComCLIP: Training-Free Compositional Image and Text Matching](https://mlsanity.com/paper/2211.13854/) (2022)
- [Towards Open-World Text-Guided Face Image Generation and Manipulation](https://mlsanity.com/paper/2104.08910/) (2021)
- [VLGrammar: Grounded Grammar Induction of Vision and Language](https://mlsanity.com/paper/2103.12975/) (2021)
- [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2306.16650/) (2023)
- [CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval](https://mlsanity.com/paper/2208.09843/) (2022)
- [Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval](https://mlsanity.com/paper/2102.04980/) (2021)
- [Distinctive Image Captioning via CLIP Guided Group Optimization](https://mlsanity.com/paper/2208.04254/) (2022)
- [Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue](https://mlsanity.com/paper/2402.03658/) (2024)
- [Geometry Attention Transformer with Position-aware LSTMs for Image Captioning](https://mlsanity.com/paper/2110.00335/) (2021)
- [Target-Augmented Shared Fusion-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2502.07391/) (2025)
- [Two-stream Hierarchical Similarity Reasoning for Image-text Matching](https://mlsanity.com/paper/2203.05349/) (2022)
- [Data Fusion for Deep Learning on Transport Mode Detection: A Case Study](https://mlsanity.com/paper/2106.05876/) (2021)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Tree-based Text-Vision BERT for Video Search in Baidu Video Advertising](https://mlsanity.com/paper/2209.08759/) (2022)
- [SCB-Dataset: A Dataset for Detecting Student and Teacher Classroom Behavior](https://mlsanity.com/paper/2304.02488/) (2023)
- [Modelling word learning and recognition using visually grounded speech](https://mlsanity.com/paper/2203.06937/) (2022)
- [Large-Scale Traffic Congestion Prediction based on Multimodal Fusion and Representation Mapping](https://mlsanity.com/paper/2208.11061/) (2022)
- [On the Cognition of Visual Question Answering Models and Human Intelligence: A Comparative Study](https://mlsanity.com/paper/2310.02528/) (2023)
- [Multi-Modal interpretable automatic video captioning](https://mlsanity.com/paper/2411.06872/) (2024)
- [PICZL: Image-based Photometric Redshifts for AGN](https://mlsanity.com/paper/2411.07305/) (2024)
- [PREMISE: Matching-based Prediction for Accurate Review Recommendation](https://mlsanity.com/paper/2505.01255/) (2025)
- [Data Transformation Strategies to Remove Heterogeneity](https://mlsanity.com/paper/2507.12677/) (2025)

## Abstract (arXiv)

In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content, and one matching CNN learning the joint representation of image and sentence. The matching CNN composes words to different semantic fragments and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results on benchmark databases of bidirectional image and sentence retrieval demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. Specifically, our proposed m-CNNs for bidirectional image and sentence retrieval on Flickr30K and Microsoft COCO databases achieve the state-of-the-art performances.