# From Cooking Recipes to Robot Task Trees -- Improving Planning Correctness and Task Efficiency by Leveraging LLMs with a Knowledge Network

arXiv: https://arxiv.org/abs/2309.09181 · HTML: https://mlsanity.com/paper/2309.09181/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose](https://mlsanity.com/paper/2007.00394/) (2020)
- [Functional Object-Oriented Network for Manipulation Learning](https://mlsanity.com/paper/1902.01537/) (2019)
- [A Survey of Knowledge Representation in Service Robotics](https://mlsanity.com/paper/1807.02192/) (2018)
- [Functional Object-Oriented Network: Construction & Expansion](https://mlsanity.com/paper/1807.02189/) (2018)
- [Approximate Task Tree Retrieval in a Knowledge Network for Robotic Cooking](https://mlsanity.com/paper/2207.03693/) (2022)
- [Joint Object and State Recognition using Language Knowledge](https://mlsanity.com/paper/1905.08843/) (2019)
- [Robot Gaining Accurate Pouring Skills through Self-Supervised Learning and Generalization](https://mlsanity.com/paper/2011.10150/) (2020)
- [Towards Robust One-shot Task Execution using Knowledge Graph Embeddings](https://mlsanity.com/paper/2105.04484/) (2021)
- [ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks](https://mlsanity.com/paper/2304.02251/) (2023)
- [Evaluating Recipes Generated from Functional Object-Oriented Network](https://mlsanity.com/paper/2106.00728/) (2021)
- [Explainable Knowledge Graph Embedding: Inference Reconciliation for Knowledge Inferences Supporting Robot Actions](https://mlsanity.com/paper/2205.01836/) (2022)
- [Long-Horizon Planning and Execution with Functional Object-Oriented Networks](https://mlsanity.com/paper/2207.05800/) (2022)
- [Learning to Pour](https://mlsanity.com/paper/1705.09021/) (2017)

## Built on by (most cited first, 9 of 9)

- [VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs](https://mlsanity.com/paper/2407.01863/) (2024)
- [On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)](https://mlsanity.com/paper/2401.02500/) (2024)
- [Generating consistent PDDL domains with Large Language Models](https://mlsanity.com/paper/2404.07751/) (2024)
- [Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability](https://mlsanity.com/paper/2506.15629/) (2025)
- [AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinement](https://mlsanity.com/paper/2502.02067/) (2025)
- [Bi-VLA: Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation](https://mlsanity.com/paper/2509.18865/) (2025)
- [Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research](https://mlsanity.com/paper/2506.12312/) (2025)
- [LEMMo-Plan: LLM-Enhanced Learning from Multi-Modal Demonstration for Planning Sequential Contact-Rich Manipulation Tasks](https://mlsanity.com/paper/2409.11863/) (2024)
- [Prompting Task Trees using Gemini: Methodologies and Insights](https://mlsanity.com/paper/2405.03671/) (2024)

## Abstract (arXiv)

Task planning for robotic cooking involves generating a sequence of actions for a robot to prepare a meal successfully. This paper introduces a novel task tree generation pipeline producing correct planning and efficient execution for cooking tasks. Our method first uses a large language model (LLM) to retrieve recipe instructions and then utilizes a fine-tuned GPT-3 to convert them into a task tree, capturing sequential and parallel dependencies among subtasks. The pipeline then mitigates the uncertainty and unreliable features of LLM outputs using task tree retrieval. We combine multiple LLM task tree outputs into a graph and perform a task tree retrieval to avoid questionable nodes and high-cost nodes to improve planning correctness and improve execution efficiency. Our evaluation results show its superior performance compared to previous works in task planning accuracy and efficiency.