mlsanityData to 7 Oct 2026

Real-World Cooking Robot System from Recipes Based on Food State Recognition Using Foundation Models and PDDL

arXiv 2410.02874 ↗First posted 3 Oct 2024cs.RO cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Although there is a growing demand for cooking behaviours as one of the expected tasks for robots, a series of cooking behaviours based on new recipe descriptions by robots in the real world has not yet been realised. In this study, we propose a robot system that integrates real-world executable robot cooking behaviour planning using the Large Language Model (LLM) and classical planning of PDDL descriptions, and food ingredient state recognition learning from a small number of data using the Vision-Language model (VLM). We succeeded in experiments in which PR2, a dual-armed wheeled robot, performed cooking from arranged new recipes in a real-world environment, and confirmed the effectiveness of the proposed system.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

18 of 34 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  7. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  8. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 45
  9. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  10. 2023Learning Object State Changes in Videos: An Open-World Perspectivecited by 36
  11. 2023Vision-Language Interpreter for Robot Task Planningcited by 29
  12. 2021How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learningcited by 15
6 more
  1. 2023Chop & Learn: Recognizing and Generating Object-State Compositionscited by 13
  2. 2022Rethinking Cooking State Recognition with Vision Transformerscited by 9
  3. 2018Classifying cooking object's state using a tuned VGG convolutional neural networkcited by 5
  4. 2023Recognition of Heat-Induced Food State Changes by Time-Series Use of Vision-Language Model for Cooking Robotcited by 5
  5. 2024Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimizationcited by 3
  6. 2020OnionBot: A System for Collaborative Computational Cookingcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  2. 2025Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progresscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Models
  2. 2026Embedding Large Language Models into Flow Controls: An Agentic Framework for Adaptive and Trustworthy Automated Cooking
  3. 2024Enabling robots to follow abstract instructions and complete complex dynamic tasks
  4. 2023Interactive Task Planning with Language Models
  5. 2024LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application
  6. 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
  7. 2023FoodLMM: A Versatile Food Assistant using Large Multi-modal Model
  8. 2026NutriBench-Kitchen: Benchmarking Embodied AI for Nutrition Management