mlsanityData to 14 Oct 2026

HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models

arXiv 2609.17771 ↗First posted 15 Sep 2026cs.RO cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Approaches to incorporating human awareness into mobile robot decision-making mainly focus on collision avoidance in low-level motion planning, often overlooking the challenges posed by human presence and high-level behavior. To address this vacancy, we present HINT-Plan, a novel approach to integrate human intention prediction into robot task planning. HINT-Plan employs Vision Language Models (VLMs) to anticipate high-level human intentions from third-person image observations, convert them into goal states, and solve joint task-planning problems. To effectively enable scene awareness in context-rich environments, we use hierarchical Scene Graphs (SGs) as high-level representations of the environment, and translate environmental topology and actionable knowledge into formal planning language to ensure executable plans. Evaluated in a photorealistic simulation, HINT-Plan achieves an overall success rate of 69.71% in joint human-robot task planning, substantially outperforming the baselines by up to 35.29%, while also reducing functional conflicts. The results show the effectiveness of explicitly incorporating inferred human intentions into formal multi-agent task planning for proactive human-aware robot decision-making.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 36 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

24 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Seeing is Believing (and Predicting): Context-Aware Multi-Human Behavior Prediction with Vision Language Models17 Dec
  2. 2025Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights1 Apr

Other references · 22

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  4. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  5. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  6. 2026Gemma 4 Technical Reportcited by 259
  7. 2018Human Action Recognition and Prediction: A Surveycited by 227
  8. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 141
  9. 2024Large Language Models Can Learn Temporal Reasoningcited by 111
  10. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  11. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
  12. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
10 more
  1. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  2. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  3. 2023PALM: Predicting Actions through Language Modelscited by 29
  4. 2019Relational Action Forecastingcited by 27
  5. 2021A Survey on Human-aware Robot Navigationcited by 24
  6. 2024Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphscited by 15
  7. 2023GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planningcited by 13
  8. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Studycited by 8
  9. 2024Towards Human Awareness in Robot Task Planning with Large Language Modelscited by 5
  10. 2025Efficient Human-Aware Task Allocation for Multi-Robot Systems in Shared Environmentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLM
  2. 2026H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model
  3. 2026HINT: Human-Intent Inception for Long-Horizon Robot Manipulation ↗
  4. 2025Towards Cognitive Collaborative Robots: Semantic-Level Integration and Explainable Control for Human-Centric Cooperation
  5. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models
  6. 2025Long-Term Planning Around Humans in Domestic Environments with 3D Scene Graphs
  7. 2025Vision-Language-Policy Model for Dynamic Robot Task Planning
  8. 2025IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interaction