# AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinement

arXiv: https://arxiv.org/abs/2502.02067 · HTML: https://mlsanity.com/paper/2502.02067/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment](https://mlsanity.com/paper/2211.08416/) (2022)
- [OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models](https://mlsanity.com/paper/2402.10670/) (2024)
- [TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation](https://mlsanity.com/paper/2402.10178/) (2024)
- [Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systems](https://mlsanity.com/paper/1804.08229/) (2018)
- [DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2404.03275/) (2024)
- [Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations](https://mlsanity.com/paper/2403.03008/) (2024)
- [Functional Object-Oriented Network for Manipulation Learning](https://mlsanity.com/paper/1902.01537/) (2019)
- [Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback](https://mlsanity.com/paper/2402.08546/) (2024)
- [Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral](https://mlsanity.com/paper/2108.11346/) (2021)
- [Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes](https://mlsanity.com/paper/2301.01751/) (2023)
- [Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks](https://mlsanity.com/paper/2102.04022/) (2021)
- [Learning Task Decomposition to Assist Humans in Competitive Programming](https://mlsanity.com/paper/2406.04604/) (2024)
- [Approximate Task Tree Retrieval in a Knowledge Network for Robotic Cooking](https://mlsanity.com/paper/2207.03693/) (2022)
- [Robot Task Planning and Situation Handling in Open Worlds](https://mlsanity.com/paper/2210.01287/) (2022)
- [From Cooking Recipes to Robot Task Trees -- Improving Planning Correctness and Task Efficiency by Leveraging LLMs with a Knowledge Network](https://mlsanity.com/paper/2309.09181/) (2023)
- [Towards Robust One-shot Task Execution using Knowledge Graph Embeddings](https://mlsanity.com/paper/2105.04484/) (2021)
- [Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.10865/) (2023)
- [Robot Task Planning Based on Large Language Model Representing Knowledge with Directed Graph Structures](https://mlsanity.com/paper/2306.05171/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order](https://mlsanity.com/paper/2601.08107/) (2026)

## Abstract (arXiv)

An embodied agent assisting humans is often asked to complete new tasks, and there may not be sufficient time or labeled examples to train the agent to perform these new tasks. Large Language Models (LLMs) trained on considerable knowledge across many domains can be used to predict a sequence of abstract actions for completing such tasks, although the agent may not be able to execute this sequence due to task-, agent-, or domain-specific constraints. Our framework addresses these challenges by leveraging the generic predictions provided by LLM and the prior domain knowledge encoded in a Knowledge Graph (KG), enabling an agent to quickly adapt to new tasks. The robot also solicits and uses human input as needed to refine its existing knowledge. Based on experimental evaluation in the context of cooking and cleaning tasks in simulation domains, we demonstrate that the interplay between LLM, KG, and human input leads to substantial performance gains compared with just using the LLM. Project websiteS: https://sssshivvvv.github.io/adaptbot/