# Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images

arXiv: https://arxiv.org/abs/1810.06553 · HTML: https://mlsanity.com/paper/1810.06553/

## Key facts

- Cited by 118 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2018.

## Built on by (most cited first, 30 of 118)

- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food](https://mlsanity.com/paper/2103.03375/) (2021)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)
- [Towards Learning Food Portion From Monocular Images With Cross-Domain Feature Adaptation](https://mlsanity.com/paper/2103.07562/) (2021)
- [FIRE: Food Image to REcipe generation](https://mlsanity.com/paper/2308.14391/) (2023)
- [MOPI-HFRS: A Multi-objective Personalized Health-aware Food Recommendation System with LLM-enhanced Interpretation](https://mlsanity.com/paper/2412.08847/) (2024)
- [Personalized Food Recommendation as Constrained Question Answering over a Large-scale Food Knowledge Graph](https://mlsanity.com/paper/2101.01775/) (2021)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [SHARE: a System for Hierarchical Assistive Recipe Editing](https://mlsanity.com/paper/2105.08185/) (2021)
- [Seeing Through Clouds in Satellite Images](https://mlsanity.com/paper/2106.08408/) (2021)
- [Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks](https://mlsanity.com/paper/2205.12396/) (2022)
- [RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models](https://mlsanity.com/paper/2407.12730/) (2024)
- [MultiSubs: A Large-scale Multimodal and Multilingual Dataset](https://mlsanity.com/paper/2103.01910/) (2021)
- [Continual learning in cross-modal retrieval](https://mlsanity.com/paper/2104.06806/) (2021)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models](https://mlsanity.com/paper/2308.03151/) (2023)
- [Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis](https://mlsanity.com/paper/2312.08592/) (2023)
- [Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning](https://mlsanity.com/paper/2405.16996/) (2024)
- [FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination](https://mlsanity.com/paper/2406.10261/) (2024)
- [Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini](https://mlsanity.com/paper/2605.27295/) (2026)
- [From Cooking Recipes to Robot Task Trees -- Improving Planning Correctness and Task Efficiency by Leveraging LLMs with a Knowledge Network](https://mlsanity.com/paper/2309.09181/) (2023)
- [GePSAn: Generative Procedure Step Anticipation in Cooking Videos](https://mlsanity.com/paper/2310.08312/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Applications of knowledge graphs for food science and industry](https://mlsanity.com/paper/2107.05869/) (2021)
- [Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineering](https://mlsanity.com/paper/2110.11592/) (2021)
- [Learning to Substitute Ingredients in Recipes](https://mlsanity.com/paper/2302.07960/) (2023)

## Abstract (arXiv)

In this paper, we introduce Recipe1M+, a new large-scale, structured corpus of over one million cooking recipes and 13 million food images. As the largest publicly available collection of recipe data, Recipe1M+ affords the ability to train high-capacity modelson aligned, multimodal data. Using these data, we train a neural network to learn a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Moreover, we demonstrate that regularization via the addition of a high-level classification objective both improves retrieval performance to rival that of humans and enables semantic vector arithmetic. We postulate that these embeddings will provide a basis for further exploration of the Recipe1M+ dataset and food and cooking in general. Code, data and models are publicly available.