mlsanityData to 7 Oct 2026

Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization

arXiv 2403.08239 ↗First posted 13 Mar 2024cs.RO cs.CV cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The state recognition of the environment and objects by robots is generally based on the judgement of the current state as a classification problem. On the other hand, state changes of food in cooking happen continuously and need to be captured not only at a certain time point but also continuously over time. In addition, the state changes of food are complex and cannot be easily described by manual programming. Therefore, we propose a method to recognize the continuous state changes of food for cooking robots through the spoken language using pre-trained large-scale vision-language models. By using models that can compute the similarity between images and texts continuously over time, we can capture the state changes of food while cooking. We also show that by adjusting the weighting of each text prompt based on fitting the similarity changes to a sigmoid function and then performing black-box optimization, more accurate and robust continuous state recognition can be achieved. We demonstrate the effectiveness and limitations of this method by performing the recognition of water boiling, butter melting, egg cooking, and onion stir-frying.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 19 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

11 of 19 references.

References · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  6. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviorscited by 12
  7. 2023VQA-based Robotic State Recognition Optimized with Genetic Algorithmcited by 10
  8. 2018Identifying Object States in Cooking-Related Imagescited by 7
  9. 2018Classifying cooking object's state using a tuned VGG convolutional neural networkcited by 5
  10. 2023Recognition of Heat-Induced Food State Changes by Time-Series Use of Vision-Language Model for Cooking Robotcited by 5
  11. 2021Cooking Object's State Identification Without Using Pretrained Modelcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  2. 2024Real-World Cooking Robot System from Recipes Based on Food State Recognition Using Foundation Models and PDDLcited by 2
  3. 2024Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking
  2. 2022Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
  3. 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
  4. 2025Efficient Prompt Tuning for Hierarchical Ingredient Recognition
  5. 2025Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking
  6. 2023FoodLMM: A Versatile Food Assistant using Large Multi-modal Model
  7. 2026FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis
  8. 2023Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model