mlsanityData to 7 Oct 2026

Creative Robot Tool Use with Large Language Models

arXiv 2310.13065 ↗First posted 19 Oct 2023cs.RO cs.AI cs.LGcited by 42

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Tool use is a hallmark of advanced intelligence, exemplified in both animal behavior and robotic capabilities. This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning. Leveraging Large Language Models (LLMs), we develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments. RoboTool incorporates four pivotal components: (i) an "Analyzer" that interprets natural language to discern key task-related concepts, (ii) a "Planner" that generates comprehensive strategies based on the language input and key concepts, (iii) a "Calculator" that computes parameters for each skill, and (iv) a "Coder" that translates these plans into executable Python code. Our results show that RoboTool can not only comprehend explicit or implicit physical constraints and environmental factors but also demonstrate creative tool use. Unlike traditional Task and Motion Planning (TAMP) methods that rely on explicit optimization, our LLM-based system offers a more flexible, efficient, and user-friendly solution for complex robotics tasks. Through extensive experiments, we validate that RoboTool is proficient in handling tasks that would otherwise be infeasible without the creative use of tools, thereby expanding the capabilities of robotic systems. Demos are available on our project page: https://creative-robotool.github.io/.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 47 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

24 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 23

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  8. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  9. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  10. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  11. 2020Integrated Task and Motion Planningcited by 281
  12. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
11 more
  1. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  2. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  3. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  4. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  5. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  6. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  7. 2014Informed RRT*: Optimal Sampling-based Path Planning Focused via Direct Sampling of an Admissible Ellipsoidal Heuristiccited by 50
  8. 2018Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervisioncited by 50
  9. 2020Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulationcited by 37
  10. 2019Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresightcited by 34
  11. 2022Leveraging Language for Accelerated Learning of Tool Manipulationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 14; 2026: 11.

42 papers.

Sort:

By area:

  1. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  3. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  4. 2024Can large language models explore in-context?cited by 75
  5. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  6. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  7. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  8. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  9. 2024An Interactive Agent Foundation Modelcited by 25
  10. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  11. 2024Programmable Motion Generation for Open-Set Motion Control Taskscited by 20
  12. 2026Large Language Model Reasoning Failurescited by 20
30 more
  1. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  2. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10
  3. 2024Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policycited by 9
  4. 2025VLMgineer: Vision Language Models as Robotic Toolsmithscited by 9
  5. 2024Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Modelscited by 8
  6. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulationcited by 8
  7. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  8. 2026ETA: A New Agentic Paradigm for Embodied Taskscited by 6
  9. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 4
  10. 2025GenSwarm: Scalable Multi-Robot Code-Policy Generation and Deployment via Language Modelscited by 4
  11. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  12. 2024RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Modelscited by 3
  13. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  14. 2024GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidancecited by 2
  15. 2025AINav: Large Language Model-Based Adaptive Interactive Navigationcited by 1
  16. 2025From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoningcited by 1
  17. 2025Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actionscited by 1
  18. 2025A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real Worldcited by 1
  19. 2026RoboWits: Unexpected Challenges for Robotic Creative Problem Solvingcited by 1
  20. 2026Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-usecited by 1
  21. 2024Growing from Exploration: A self-exploring framework for robots based on foundation modelscited by 0
  22. 2024Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neurosciencecited by 0
  23. 2025Prolonging Tool Life: Learning Skillful Use of General-purpose Tools through Lifespan-guided Reinforcement Learningcited by 0
  24. 2025Toward Accurate Long-Horizon Robotic Manipulation: Language-to-Action with Foundation Models via Scene Graphscited by 0
  25. 2026RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Modelscited by 0
  26. 2026Long-Term Memory for VLA-based Agents in Open-World Task Executioncited by 0
  27. 2026Creative Robot Tool Use by Counterfactual Reasoningcited by 0
  28. 2026Beyond APIs: Probing the Limits of MLLMs in Physical Tool Usecited by 0
  29. 2026GROW2: Grounding Which and Where for Robot Tool Usecited by 0
  30. 2026SimEX: Simulation-Integrated Robotics AutoResearchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Constrained Natural Language Action Planning for Resilient Embodied Systems
  2. 2023Conditionally Combining Robot Skills using Large Language Models
  3. 2025A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
  4. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs
  5. 2024PLATO: Planning with LLMs and Affordances for Tool Manipulation
  6. 2025Leveraging Pre-trained Large Language Models with Refined Prompting for Online Task and Motion Planning
  7. 2025Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generation
  8. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition