# Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA

arXiv: https://arxiv.org/abs/1911.12763 · HTML: https://mlsanity.com/paper/1911.12763/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 7 of 7)

- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [Learning TFIDF Enhanced Joint Embedding for Recipe-Image Cross-Modal Retrieval Service](https://mlsanity.com/paper/2108.00724/) (2021)
- [Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2204.09730/) (2022)
- [Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval](https://mlsanity.com/paper/2207.14428/) (2022)
- [Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2212.04267/) (2022)
- [MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval](https://mlsanity.com/paper/2305.11327/) (2023)

## Abstract (arXiv)

We propose a novel non-parametric method for cross-modal recipe retrieval which is applied on top of precomputed image and text embeddings. By combining our method with standard approaches for building image and text encoders, trained independently with a self-supervised classification objective, we create a baseline model which outperforms most existing methods on a challenging image-to-recipe task. We also use our method for comparing image and text encoders trained using different modern approaches, thus addressing the issues hindering the development of novel methods for cross-modal recipe retrieval. We demonstrate how to use the insights from model comparison and extend our baseline model with standard triplet loss that improves state-of-the-art on the Recipe1M dataset by a large margin, while using only precomputed features and with much less complexity than existing methods. Further, our approach readily generalizes beyond recipe retrieval to other challenging domains, achieving state-of-the-art performance on Politics and GoodNews cross-modal retrieval tasks.