# Enhancing Recipe Retrieval with Foundation Models: A Data Augmentation Perspective

arXiv: https://arxiv.org/abs/2312.04763 · HTML: https://mlsanity.com/paper/2312.04763/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2023.

## Built on by (most cited first, 10 of 10)

- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [FoodLMM: A Versatile Food Assistant using Large Multi-modal Model](https://mlsanity.com/paper/2312.14991/) (2023)
- [Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion](https://mlsanity.com/paper/2505.08747/) (2025)
- [CookingDiffusion: Cooking Procedural Image Generation with Stable Diffusion](https://mlsanity.com/paper/2501.09042/) (2025)
- [Retrieval Augmented Recipe Generation](https://mlsanity.com/paper/2411.08715/) (2024)
- [LLM-Driven Completeness and Consistency Evaluation for Cultural Heritage Data Augmentation in Cross-Modal Retrieval](https://mlsanity.com/paper/2511.06268/) (2025)
- [Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning](https://mlsanity.com/paper/2511.13351/) (2025)
- [Efficient Test-Time Retrieval Augmented Generation](https://mlsanity.com/paper/2511.01059/) (2025)
- [SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding](https://mlsanity.com/paper/2604.15628/) (2026)
- [ReGraph: Learning to Generate Recipe Graphs from Food Images](https://mlsanity.com/paper/2608.06917/) (2026)

## Abstract (arXiv)

Learning recipe and food image representation in common embedding space is non-trivial but crucial for cross-modal recipe retrieval. In this paper, we propose a new perspective for this problem by utilizing foundation models for data augmentation. Leveraging on the remarkable capabilities of foundation models (i.e., Llama2 and SAM), we propose to augment recipe and food image by extracting alignable information related to the counterpart. Specifically, Llama2 is employed to generate a textual description from the recipe, aiming to capture the visual cues of a food image, and SAM is used to produce image segments that correspond to key ingredients in the recipe. To make full use of the augmented data, we introduce Data Augmented Retrieval framework (DAR) to enhance recipe and image representation learning for cross-modal retrieval. We first inject adapter layers to pre-trained CLIP model to reduce computation cost rather than fully fine-tuning all the parameters. In addition, multi-level circle loss is proposed to align the original and augmented data pairs, which assigns different penalties for positive and negative pairs. On the Recipe1M dataset, our DAR outperforms all existing methods by a large margin. Extensive ablation studies validate the effectiveness of each component of DAR.