# Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings

arXiv: https://arxiv.org/abs/1804.11146 · HTML: https://mlsanity.com/paper/1804.11146/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2018.

## Built on by (most cited first, 30 of 35)

- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Robust and Decomposable Average Precision for Image Retrieval](https://mlsanity.com/paper/2110.01445/) (2021)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks](https://mlsanity.com/paper/2205.12396/) (2022)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [Optimization of Rank Losses for Image Retrieval](https://mlsanity.com/paper/2309.08250/) (2023)
- [Learnable Pillar-based Re-ranking for Image-Text Retrieval](https://mlsanity.com/paper/2304.12570/) (2023)
- [Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineering](https://mlsanity.com/paper/2110.11592/) (2021)
- [CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval](https://mlsanity.com/paper/2102.02547/) (2021)
- [Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring](https://mlsanity.com/paper/2107.00372/) (2021)
- [Learning TFIDF Enhanced Joint Embedding for Recipe-Image Cross-Modal Retrieval Service](https://mlsanity.com/paper/2108.00724/) (2021)
- [Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2204.09730/) (2022)
- [Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval](https://mlsanity.com/paper/2207.14428/) (2022)
- [The Multimodal And Modular Ai Chef: Complex Recipe Generation From Imagery](https://mlsanity.com/paper/2304.02016/) (2023)
- [Learning Joint Embedding with Modality Alignments for Cross-Modal Retrieval of Recipes and Food Images](https://mlsanity.com/paper/2108.03788/) (2021)
- [A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval](https://mlsanity.com/paper/2201.02772/) (2022)
- [RecipeMind: Guiding Ingredient Choices from Food Pairing to Recipe Completion using Cascaded Set Transformer](https://mlsanity.com/paper/2210.10628/) (2022)
- [Cross-lingual Adaptation for Recipe Retrieval with Mixup](https://mlsanity.com/paper/2205.03891/) (2022)
- [Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2212.04267/) (2022)
- [Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study](https://mlsanity.com/paper/2301.05174/) (2023)
- [Cross-Modal Entity Matching for Visually Rich Documents](https://mlsanity.com/paper/2303.00720/) (2023)
- [Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training](https://mlsanity.com/paper/2205.04948/) (2022)
- [Facing the Void: Overcoming Missing Data in Multi-View Imagery](https://mlsanity.com/paper/2205.10592/) (2022)
- [QMRNet: Quality Metric Regression for EO Image Quality Assessment and Super-Resolution](https://mlsanity.com/paper/2210.06618/) (2022)
- [MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval](https://mlsanity.com/paper/2305.11327/) (2023)
- [Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval](https://mlsanity.com/paper/2510.20393/) (2025)

## Abstract (arXiv)

Designing powerful tools that support cooking activities has rapidly gained popularity due to the massive amounts of available data, as well as recent advances in machine learning that are capable of analyzing them. In this paper, we propose a cross-modal retrieval model aligning visual and textual data (like pictures of dishes and their recipes) in a shared representation space. We describe an effective learning scheme, capable of tackling large-scale problems, and validate it on the Recipe1M dataset containing nearly 1 million picture-recipe pairs. We show the effectiveness of our approach regarding previous state-of-the-art models and present qualitative results over computational cooking use cases.