# Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning

arXiv: https://arxiv.org/abs/2103.13061 · HTML: https://mlsanity.com/paper/2103.13061/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models](https://mlsanity.com/paper/1711.06420/) (2017)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [ISIA Food-500: A Dataset for Large-Scale Food Recognition via Stacked Global-Local Attention Network](https://mlsanity.com/paper/2008.05655/) (2020)
- [Learning Video Representations from Textual Web Supervision](https://mlsanity.com/paper/2007.14937/) (2020)
- [Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images](https://mlsanity.com/paper/1905.01273/) (2019)
- [Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings](https://mlsanity.com/paper/1804.11146/) (2018)
- [ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognition](https://mlsanity.com/paper/1705.02743/) (2017)
- [Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism](https://mlsanity.com/paper/2003.03955/) (2020)
- [DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment](https://mlsanity.com/paper/1606.05675/) (2016)
- [Structure-Aware Generation Network for Recipe Generation from Images](https://mlsanity.com/paper/2009.00944/) (2020)
- [MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model](https://mlsanity.com/paper/2004.01095/) (2020)
- [Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health](https://mlsanity.com/paper/1702.06318/) (2017)
- [Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA](https://mlsanity.com/paper/1911.12763/) (2019)
- [Procedural Reasoning Networks for Understanding Multimodal Procedures](https://mlsanity.com/paper/1909.08859/) (2019)
- [Picture-to-Amount (PITA): Predicting Relative Ingredient Amounts from Food Images](https://mlsanity.com/paper/2010.08727/) (2020)

## Built on by (most cited first, 28 of 28)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment](https://mlsanity.com/paper/2307.03373/) (2023)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks](https://mlsanity.com/paper/2205.12396/) (2022)
- [Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis](https://mlsanity.com/paper/2312.08592/) (2023)
- [FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination](https://mlsanity.com/paper/2406.10261/) (2024)
- [Learning to Substitute Ingredients in Recipes](https://mlsanity.com/paper/2302.07960/) (2023)
- [Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2204.09730/) (2022)
- [Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval](https://mlsanity.com/paper/2207.14428/) (2022)
- [CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion](https://mlsanity.com/paper/2501.09042/) (2025)
- [Cross-lingual Adaptation for Recipe Retrieval with Mixup](https://mlsanity.com/paper/2205.03891/) (2022)
- [Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2212.04267/) (2022)
- [Retrieval Augmented Recipe Generation](https://mlsanity.com/paper/2411.08715/) (2024)
- [COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking](https://mlsanity.com/paper/2504.01321/) (2025)
- [A Closed-Loop Multi-Agent System Driven by LLMs for Meal-Level Personalized Nutrition Management](https://mlsanity.com/paper/2601.04491/) (2026)
- [Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training](https://mlsanity.com/paper/2205.04948/) (2022)
- [MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval](https://mlsanity.com/paper/2305.11327/) (2023)
- [Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval](https://mlsanity.com/paper/2510.20393/) (2025)
- [Spectral Discrepancy and Cross-modal Semantic Consistency Learning for Object Detection in Hyperspectral Image](https://mlsanity.com/paper/2512.18245/) (2025)
- [RecipeSnap -- a lightweight image-to-recipe model](https://mlsanity.com/paper/2205.02141/) (2022)
- [Predefined domain specific embeddings of food concepts and recipes: A case study on heterogeneous recipe datasets](https://mlsanity.com/paper/2302.01005/) (2023)
- [Efficient Test-Time Retrieval Augmented Generation](https://mlsanity.com/paper/2511.01059/) (2025)
- [Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval](https://mlsanity.com/paper/2511.15201/) (2025)
- [LLMs-based Augmentation for Domain Adaptation in Long-tailed Food Datasets](https://mlsanity.com/paper/2511.16037/) (2025)
- [Fusing Semantic, Lexical, and Domain Perspectives for Recipe Similarity Estimation](https://mlsanity.com/paper/2603.09688/) (2026)
- [SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding](https://mlsanity.com/paper/2604.15628/) (2026)
- [RFHNet: Relational and Frequency-Aware Hashing Network for Large-Scale Fine-Grained Food Image Retrieval](https://mlsanity.com/paper/2607.06148/) (2026)

## Abstract (arXiv)

Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives, as well as the availability of vast amounts of digital cooking recipes and food images to train machine learning models. In this work, we revisit existing approaches for cross-modal recipe retrieval and propose a simplified end-to-end model based on well established and high performing encoders for text and images. We introduce a hierarchical recipe Transformer which attentively encodes individual recipe components (titles, ingredients and instructions). Further, we propose a self-supervised loss function computed on top of pairs of individual recipe components, which is able to leverage semantic relationships within recipes, and enables training using both image-recipe and recipe-only samples. We conduct a thorough analysis and ablation studies to validate our design choices. As a result, our proposed method achieves state-of-the-art performance in the cross-modal recipe retrieval task on the Recipe1M dataset. We make code and models publicly available.