mlsanityData to 7 Oct 2026

Learning Program Representations for Food Images and Cooking Recipes

arXiv 2203.16071 ↗First posted 30 Mar 2022cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

In this paper, we are interested in modeling a how-to instructional procedure, such as a cooking recipe, with a meaningful and rich high-level representation. Specifically, we propose to represent cooking recipes and food images as cooking programs. Programs provide a structured representation of the task, capturing cooking semantics and sequential relationships of actions in the form of a graph. This allows them to be easily manipulated by users and executed by agents. To this end, we build a model that is trained to learn a joint embedding between recipes and food images via self-supervision and jointly generate a program from this embedding as a sequence. To validate our idea, we crowdsource programs for cooking recipes and show that: (a) projecting the image-recipe embeddings into programs leads to better cross-modal retrieval results; (b) generating programs from images leads to better recognition results compared to predicting raw cooking instructions; and (c) we can generate food images by manipulating programs via optimizing the latent code of a GAN. Code, data, and models are available online.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 40 of its 66 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

40 of 66 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  9. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2020Quantifying Attention Flow in Transformerscited by 759
  12. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
28 more
  1. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  2. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  3. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  4. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  5. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  6. 2017CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noisecited by 181
  7. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  8. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  9. 2018Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Imagescited by 118
  10. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  11. 2021Paint by Wordcited by 107
  12. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  13. 2018Inverse Cooking: Recipe Generation from Food Imagescited by 47
  14. 2020ISIA Food-500: A Dataset for Large-Scale Food Recognition via Stacked Global-Local Attention Networkcited by 47
  15. 2019Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Imagescited by 38
  16. 2018Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddingscited by 35
  17. 2019FoodX-251: A Dataset for Fine-grained Food Classificationcited by 33
  18. 2018Finding beans in burgers: Deep semantic-visual embedding with localizationcited by 28
  19. 2021Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learningcited by 28
  20. 2017ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognitioncited by 21
  21. 2019How to make a pizza: Learning a compositional layer-based GAN modelcited by 21
  22. 2020Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanismcited by 20
  23. 2016DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessmentcited by 19
  24. 2020MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Modelcited by 16
  25. 2020CookGAN: Meal Image Synthesis from Ingredientscited by 14
  26. 2020A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videoscited by 11
  27. 2019Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTAcited by 7
  28. 2021CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrievalcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 3; 2025: 4; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023FIRE: Food Image to REcipe generationcited by 18
  2. 2024FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotationcited by 11
  3. 2022Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flowscited by 8
  4. 2022Vision and Structured-Language Pretraining for Cross-Modal Food Retrievalcited by 3
  5. 2023Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipescited by 3
  6. 2022Recipe Generation from Unsegmented Cooking Videoscited by 2
  7. 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generationcited by 2
  8. 2023MALM: Mask Augmentation based Local Matching for Food-Recipe Retrievalcited by 1
  9. 2025Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrievalcited by 1
  10. 2023Towards Flow Graph Prediction of Open-Domain Procedural Textscited by 0
  11. 2024Unsupervised Learning of Graph from Recipescited by 0
  12. 2025A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Taskcited by 0
4 more
  1. 2025Towards an Action-Centric Ontology for Cooking Procedures Using Temporal Graphscited by 0
  2. 2025Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrievalcited by 0
  3. 2026SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embeddingcited by 0
  4. 2026ReGraph: Learning to Generate Recipe Graphs from Food Imagescited by 0