# Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images

arXiv: https://arxiv.org/abs/1905.01273 · HTML: https://mlsanity.com/paper/1905.01273/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2019.

## Built on by (most cited first, 30 of 39)

- [Large Scale Visual Food Recognition](https://mlsanity.com/paper/2103.16107/) (2021)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Cycle-Consistent Inverse GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.01361/) (2021)
- [Learning Structural Representations for Recipe Generation and Food Retrieval](https://mlsanity.com/paper/2110.01209/) (2021)
- [Learning Program Representations for Food Images and Cooking Recipes](https://mlsanity.com/paper/2203.16071/) (2022)
- [Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks](https://mlsanity.com/paper/2205.12396/) (2022)
- [ASMR: Learning Attribute-Based Person Search with Adaptive Semantic Margin Regularizer](https://mlsanity.com/paper/2108.04533/) (2021)
- [ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation](https://mlsanity.com/paper/2409.12010/) (2024)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [Metric Learning for Anti-Compression Facial Forgery Detection](https://mlsanity.com/paper/2103.08397/) (2021)
- [Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineering](https://mlsanity.com/paper/2110.11592/) (2021)
- [CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval](https://mlsanity.com/paper/2102.02547/) (2021)
- [Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring](https://mlsanity.com/paper/2107.00372/) (2021)
- [Learning TFIDF Enhanced Joint Embedding for Recipe-Image Cross-Modal Retrieval Service](https://mlsanity.com/paper/2108.00724/) (2021)
- [Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2204.09730/) (2022)
- [Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval](https://mlsanity.com/paper/2207.14428/) (2022)
- [Learning Joint Embedding with Modality Alignments for Cross-Modal Retrieval of Recipes and Food Images](https://mlsanity.com/paper/2108.03788/) (2021)
- [Cross-Modal Graph with Meta Concepts for Video Captioning](https://mlsanity.com/paper/2108.06458/) (2021)
- [Cross-lingual Adaptation for Recipe Retrieval with Mixup](https://mlsanity.com/paper/2205.03891/) (2022)
- [Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval](https://mlsanity.com/paper/2212.04267/) (2022)
- [Synthesizing Knowledge-enhanced Features for Real-world Zero-shot Food Detection](https://mlsanity.com/paper/2402.09242/) (2024)
- [Retrieval Augmented Recipe Generation](https://mlsanity.com/paper/2411.08715/) (2024)
- [Cross-modal Image Retrieval with Deep Mutual Information Maximization](https://mlsanity.com/paper/2103.06032/) (2021)
- [Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method](https://mlsanity.com/paper/2203.15778/) (2022)
- [Self-Augmented Multi-Modal Feature Embedding](https://mlsanity.com/paper/2103.04731/) (2021)
- [Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training](https://mlsanity.com/paper/2205.04948/) (2022)
- [SLAN: Self-Locator Aided Network for Cross-Modal Understanding](https://mlsanity.com/paper/2211.16208/) (2022)
- [MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval](https://mlsanity.com/paper/2305.11327/) (2023)
- [Shape-Preserving Generation of Food Images for Automatic Dietary Assessment](https://mlsanity.com/paper/2408.13358/) (2024)
- [Khana: A Comprehensive Indian Cuisine Dataset](https://mlsanity.com/paper/2509.06006/) (2025)

## Abstract (arXiv)

Food computing is playing an increasingly important role in human daily life, and has found tremendous applications in guiding human behavior towards smart food consumption and healthy lifestyle. An important task under the food-computing umbrella is retrieval, which is particularly helpful for health related applications, where we are interested in retrieving important information about food (e.g., ingredients, nutrition, etc.). In this paper, we investigate an open research task of cross-modal retrieval between cooking recipes and food images, and propose a novel framework Adversarial Cross-Modal Embedding (ACME) to resolve the cross-modal retrieval task in food domains. Specifically, the goal is to learn a common embedding feature space between the two modalities, in which our approach consists of several novel ideas: (i) learning by using a new triplet loss scheme together with an effective sampling strategy, (ii) imposing modality alignment using an adversarial learning strategy, and (iii) imposing cross-modal translation consistency such that the embedding of one modality is able to recover some important information of corresponding instances in the other modality. ACME achieves the state-of-the-art performance on the benchmark Recipe1M dataset, validating the efficacy of the proposed technique.