# GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning

arXiv: https://arxiv.org/abs/2310.20034 · HTML: https://mlsanity.com/paper/2310.20034/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2023.

## Built on by (most cited first, 13 of 13)

- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)](https://mlsanity.com/paper/2401.02500/) (2024)
- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [Large Language Models are Zero-Shot Recognizers for Activities of Daily Living](https://mlsanity.com/paper/2407.01238/) (2024)
- [Using Large Language Models to Compare Explainable Models for Smart Home Human Activity Recognition](https://mlsanity.com/paper/2408.06352/) (2024)
- [Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs](https://mlsanity.com/paper/2408.06520/) (2024)
- [Dual-AEB: Synergizing Rule-Based and Multimodal Large Language Models for Effective Emergency Braking](https://mlsanity.com/paper/2410.08616/) (2024)
- [Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task](https://mlsanity.com/paper/2404.08424/) (2024)
- [Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights](https://mlsanity.com/paper/2504.00839/) (2025)
- [Seeing is Believing (and Predicting): Context-Aware Multi-Human Behavior Prediction with Vision Language Models](https://mlsanity.com/paper/2512.15957/) (2025)
- [ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models](https://mlsanity.com/paper/2403.06586/) (2024)
- [SafeStep: AI-powered Travel Assistance for Elderly People with Frailty or Dementia](https://mlsanity.com/paper/2607.21156/) (2026)
- [HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models](https://mlsanity.com/paper/2609.17771/) (2026)

## Abstract (arXiv)

A robot in a human-centric environment needs to account for the human's intent and future motion in its task and motion planning to ensure safe and effective operation. This requires symbolic reasoning about probable future actions and the ability to tie these actions to specific locations in the physical environment. While one can train behavioral models capable of predicting human motion from past activities, this approach requires large amounts of data to achieve acceptable long-horizon predictions. More importantly, the resulting models are constrained to specific data formats and modalities. Moreover, connecting predictions from such models to the environment at hand to ensure the applicability of these predictions is an unsolved problem. We present a system that utilizes a Large Language Model (LLM) to infer a human's next actions from a range of modalities without fine-tuning. A novel aspect of our system that is critical to robotics applications is that it links the predicted actions to specific locations in a semantic map of the environment. Our method leverages the fact that LLMs, trained on a vast corpus of text describing typical human behaviors, encode substantial world knowledge, including probable sequences of human actions and activities. We demonstrate how these localized activity predictions can be incorporated in a human-aware task planner for an assistive robot to reduce the occurrences of undesirable human-robot interactions by 29.2% on average.