# Learning to Substitute Ingredients in Recipes

arXiv: https://arxiv.org/abs/2302.07960 · HTML: https://mlsanity.com/paper/2302.07960/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 65 references that are in the map.
- First posted on arXiv 15 Feb 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings](https://mlsanity.com/paper/2006.13009/) (2020)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks](https://mlsanity.com/paper/2102.05034/) (2021)
- [Enriching Pre-trained Language Model with Entity Information for Relation Classification](https://mlsanity.com/paper/1905.08284/) (2019)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [Knowledge Base Completion: Baselines Strike Back](https://mlsanity.com/paper/1705.10744/) (2017)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Structure-Aware Generation Network for Recipe Generation from Images](https://mlsanity.com/paper/2009.00944/) (2020)
- [Personalized Food Recommendation as Constrained Question Answering over a Large-scale Food Knowledge Graph](https://mlsanity.com/paper/2101.01775/) (2021)
- [The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures](https://mlsanity.com/paper/2104.10640/) (2021)
- [SHARE: a System for Hierarchical Assistive Recipe Editing](https://mlsanity.com/paper/2105.08185/) (2021)
- [CookGAN: Meal Image Synthesis from Ingredients](https://mlsanity.com/paper/2002.11493/) (2020)
- [Elucidating image-to-set prediction: An analysis of models, losses and datasets](https://mlsanity.com/paper/1904.05709/) (2019)

## Built on by (most cited first, 8 of 8)

- [LLaVA-Chef: A Multi-modal Generative Model for Food Recipes](https://mlsanity.com/paper/2408.16889/) (2024)
- [Generating Illustrated Instructions](https://mlsanity.com/paper/2312.04552/) (2023)
- [GLEN-Bench: A Graph-Language based Benchmark for Nutritional Health](https://mlsanity.com/paper/2601.18106/) (2026)
- [Large Language Models for Ingredient Substitution in Food Recipes using Supervised Fine-tuning and Direct Preference Optimization](https://mlsanity.com/paper/2412.04922/) (2024)
- [Optimizing Ingredient Substitution Using Large Language Models to Enhance Phytochemical Content in Recipes](https://mlsanity.com/paper/2409.08792/) (2024)
- ["Stop replacing salt with sugar!'': Towards Intuitive Human-Agent Teaching](https://mlsanity.com/paper/2509.24651/) (2025)
- [Fuzzy Categorical Planning: Autonomous Goal Satisfaction with Graded Semantic Constraints](https://mlsanity.com/paper/2601.20021/) (2026)
- [FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training](https://mlsanity.com/paper/2608.20574/) (2026)

## Abstract (arXiv)

Recipe personalization through ingredient substitution has the potential to help people meet their dietary needs and preferences, avoid potential allergens, and ease culinary exploration in everyone's kitchen. To address ingredient substitution, we build a benchmark, composed of a dataset of substitution pairs with standardized splits, evaluation metrics, and baselines. We further introduce Graph-based Ingredient Substitution Module (GISMo), a novel model that leverages the context of a recipe as well as generic ingredient relational information encoded within a graph to rank plausible substitutions. We show through comprehensive experimental validation that GISMo surpasses the best performing baseline by a large margin in terms of mean reciprocal rank. Finally, we highlight the benefits of GISMo by integrating it in an improved image-to-recipe generation pipeline, enabling recipe personalization through user intervention. Quantitative and qualitative results show the efficacy of our proposed system, paving the road towards truly personalized cooking and tasting experiences.