# Towards Human Awareness in Robot Task Planning with Large Language Models

arXiv: https://arxiv.org/abs/2404.11267 · HTML: https://mlsanity.com/paper/2404.11267/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2024.

## Built on by (most cited first, 5 of 5)

- [REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2503.22122/) (2025)
- [AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robots](https://mlsanity.com/paper/2409.11905/) (2024)
- [Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights](https://mlsanity.com/paper/2504.00839/) (2025)
- [Defining and Monitoring Complex Robot Activities via LLMs and Symbolic Reasoning](https://mlsanity.com/paper/2509.16006/) (2025)
- [HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models](https://mlsanity.com/paper/2609.17771/) (2026)

## Abstract (arXiv)

The recent breakthroughs in the research on Large Language Models (LLMs) have triggered a transformation across several research domains. Notably, the integration of LLMs has greatly enhanced performance in robot Task And Motion Planning (TAMP). However, previous approaches often neglect the consideration of dynamic environments, i.e., the presence of dynamic objects such as humans. In this paper, we propose a novel approach to address this gap by incorporating human awareness into LLM-based robot task planning. To obtain an effective representation of the dynamic environment, our approach integrates humans' information into a hierarchical scene graph. To ensure the plan's executability, we leverage LLMs to ground the environmental topology and actionable knowledge into formal planning language. Most importantly, we use LLMs to predict future human activities and plan tasks for the robot considering the predictions. Our contribution facilitates the development of integrating human awareness into LLM-driven robot task planning, and paves the way for proactive robot decision-making in dynamic environments.