# A Human-in-the-loop Approach to Robot Action Replanning through LLM Common-Sense Reasoning

arXiv: https://arxiv.org/abs/2507.20870 · HTML: https://mlsanity.com/paper/2507.20870/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 8 of 8)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks](https://mlsanity.com/paper/2210.04476/) (2022)
- [Zero-shot Task Adaptation using Natural Language](https://mlsanity.com/paper/2106.02972/) (2021)

## Built on by (most cited first, 3 of 3)

- [Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaboration](https://mlsanity.com/paper/2511.08732/) (2025)
- [IROSA: Interactive Robot Skill Adaptation using Natural Language](https://mlsanity.com/paper/2603.03897/) (2026)
- [Social Structure Matters in 3D Human-Human Interaction Generation](https://mlsanity.com/paper/2606.24255/) (2026)

## Abstract (arXiv)

To facilitate the wider adoption of robotics, accessible programming tools are required for non-experts. Observational learning enables intuitive human skills transfer through hands-on demonstrations, but relying solely on visual input can be inefficient in terms of scalability and failure mitigation, especially when based on a single demonstration. This paper presents a human-in-the-loop method for enhancing the robot execution plan, automatically generated based on a single RGB video, with natural language input to a Large Language Model (LLM). By including user-specified goals or critical task aspects and exploiting the LLM common-sense reasoning, the system adjusts the vision-based plan to prevent potential failures and adapts it based on the received instructions. Experiments demonstrated the framework intuitiveness and effectiveness in correcting vision-derived errors and adapting plans without requiring additional demonstrations. Moreover, interactive plan refinement and hallucination corrections promoted system robustness.