# Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models

arXiv: https://arxiv.org/abs/1711.06420 · HTML: https://mlsanity.com/paper/1711.06420/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2017.

## Built on by (most cited first, 30 of 53)

- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss](https://mlsanity.com/paper/2109.04290/) (2021)
- [Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification](https://mlsanity.com/paper/2107.12666/) (2021)
- [DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents](https://mlsanity.com/paper/2101.11796/) (2021)
- [LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval](https://mlsanity.com/paper/2103.08784/) (2021)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO](https://mlsanity.com/paper/2204.03359/) (2022)
- [Improving Cross-Modal Retrieval with Set of Diverse Embeddings](https://mlsanity.com/paper/2211.16761/) (2022)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design](https://mlsanity.com/paper/2106.13058/) (2021)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Imagination-Augmented Natural Language Understanding](https://mlsanity.com/paper/2204.08535/) (2022)
- [Vision-Language Dataset Distillation](https://mlsanity.com/paper/2308.07545/) (2023)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control](https://mlsanity.com/paper/2402.17535/) (2024)
- [DR-GAN: Distribution Regularization for Text-to-Image Generation](https://mlsanity.com/paper/2204.07945/) (2022)
- [Plug-and-Play Regulators for Image-Text Matching](https://mlsanity.com/paper/2303.13371/) (2023)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching](https://mlsanity.com/paper/2104.10406/) (2021)
- [Cross-Modal Generative Augmentation for Visual Question Answering](https://mlsanity.com/paper/2105.04780/) (2021)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [PHyCLIP: ℓ1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning](https://mlsanity.com/paper/2510.08919/) (2025)
- [VLDeformer: Vision-Language Decomposed Transformer for Fast Cross-Modal Retrieval](https://mlsanity.com/paper/2110.11338/) (2021)
- [Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition](https://mlsanity.com/paper/2202.07901/) (2022)
- [LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives](https://mlsanity.com/paper/2203.01445/) (2022)
- [What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?](https://mlsanity.com/paper/2310.06383/) (2023)
- [From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification](https://mlsanity.com/paper/2408.00096/) (2024)
- [Aligning Information Capacity Between Vision and Language via Dense-to-Sparse Feature Distillation for Image-Text Matching](https://mlsanity.com/paper/2503.14953/) (2025)

## Abstract (arXiv)

Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for multi-modal data is crucial for the cross-modal retrieval performance. Unlike existing image-text retrieval approaches that embed image-text pairs as single feature vectors in a common representational space, we propose to incorporate generative processes into the cross-modal feature embedding, through which we are able to learn not only the global abstract features but also the local grounded features. Extensive experiments show that our framework can well match images and sentences with complex content, and achieve the state-of-the-art cross-modal retrieval results on MSCOCO dataset.