# Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model

arXiv: https://arxiv.org/abs/2308.15684 · HTML: https://mlsanity.com/paper/2308.15684/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers](https://mlsanity.com/paper/2006.03555/) (2020)
- [Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects](https://mlsanity.com/paper/2205.05960/) (2022)
- [VQA-based Robotic State Recognition Optimized with Genetic Algorithm](https://mlsanity.com/paper/2303.05052/) (2023)
- [Deep Predictive Learning: Motion Learning Concept inspired by Cognitive Robotics](https://mlsanity.com/paper/2306.14714/) (2023)
- [Embodying Pre-Trained Word Embeddings Through Robot Actions](https://mlsanity.com/paper/2104.08521/) (2021)
- [Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data](https://mlsanity.com/paper/2203.04218/) (2022)

## Built on by (most cited first, 3 of 3)

- [A Survey on Robotics with Foundation Models: toward Embodied AI](https://mlsanity.com/paper/2402.02385/) (2024)
- [ReplanVLM: Replanning Robotic Tasks with Visual Language Models](https://mlsanity.com/paper/2407.21762/) (2024)
- [Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework](https://mlsanity.com/paper/2411.01639/) (2024)

## Abstract (arXiv)

The application of the Large Language Model (LLM) to robot action planning has been actively studied. The instructions given to the LLM by natural language may include ambiguity and lack of information depending on the task context. It is possible to adjust the output of LLM by making the instruction input more detailed; however, the design cost is high. In this paper, we propose the interactive robot action planning method that allows the LLM to analyze and gather missing information by asking questions to humans. The method can minimize the design cost of generating precise robot instructions. We demonstrated the effectiveness of our method through concrete examples in cooking tasks. However, our experiments also revealed challenges in robot action planning with LLM, such as asking unimportant questions and assuming crucial information without asking. Shedding light on these issues provides valuable insights for future research on utilizing LLM for robotics.