# Plan-and-Act using Large Language Models for Interactive Agreement

arXiv: https://arxiv.org/abs/2504.01252 · HTML: https://mlsanity.com/paper/2504.01252/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023)
- [ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application](https://mlsanity.com/paper/2304.03893/) (2023)
- [Understanding Large-Language Model (LLM)-powered Human-Robot Interaction](https://mlsanity.com/paper/2401.03217/) (2024)
- [Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?](https://mlsanity.com/paper/2401.05302/) (2024)
- [LaMI: Large Language Models for Multi-Modal Human-Robot Interaction](https://mlsanity.com/paper/2401.15174/) (2024)
- [Towards Robust and Unconstrained Full Range of Rotation Head Pose Estimation](https://mlsanity.com/paper/2309.07654/) (2023)
- [Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models](https://mlsanity.com/paper/2308.16529/) (2023)
- [A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations](https://mlsanity.com/paper/2008.01513/) (2020)
- [Semantic constraints to represent common sense required in household actions for multi-modal Learning-from-observation robot](https://mlsanity.com/paper/2103.02201/) (2021)
- [Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models](https://mlsanity.com/paper/2503.15491/) (2025)
- [Designing Library of Skill-Agents for Hardware-Level Reusability](https://mlsanity.com/paper/2403.02316/) (2024)

## Built on by (most cited first, 1 of 1)

- [SLM-Based Agentic AI with P-C-G: Optimized for Korean Tool Use](https://mlsanity.com/paper/2509.19369/) (2025)

## Abstract (arXiv)

Recent large language models (LLMs) are capable of planning robot actions. In this paper, we explore how LLMs can be used for planning actions with tasks involving situational human-robot interaction (HRI). A key problem of applying LLMs in situational HRI is balancing between "respecting the current human's activity" and "prioritizing the robot's task," as well as understanding the timing of when to use the LLM to generate an action plan. In this paper, we propose a necessary plan-and-act skill design to solve the above problems. We show that a critical factor for enabling a robot to switch between passive / active interaction behavior is to provide the LLM with an action text about the current robot's action. We also show that a second-stage question to the LLM (about the next timing to call the LLM) is necessary for planning actions at an appropriate timing. The skill design is applied to an Engage skill and is tested on four distinct interaction scenarios. We show that by using the skill design, LLMs can be leveraged to easily scale to different HRI scenarios with a reasonable success rate reaching 90% on the test scenarios.