# Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task

arXiv: https://arxiv.org/abs/2404.08424 · HTML: https://mlsanity.com/paper/2404.08424/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [BlazePose: On-device Real-time Body Pose tracking](https://mlsanity.com/paper/2006.10204/) (2020)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)](https://mlsanity.com/paper/2302.06706/) (2023)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination](https://mlsanity.com/paper/2312.15224/) (2023)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [LaMI: Large Language Models for Multi-Modal Human-Robot Interaction](https://mlsanity.com/paper/2401.15174/) (2024)
- [NICOL: A Neuro-inspired Collaborative Semi-humanoid Robot that Bridges Social Interaction and Reliable Manipulation](https://mlsanity.com/paper/2305.08528/) (2023)
- [GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning](https://mlsanity.com/paper/2310.20034/) (2023)
- [From Text to Motion: Grounding GPT-4 in a Humanoid Robot "Alter3"](https://mlsanity.com/paper/2312.06571/) (2023)
- [Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models](https://mlsanity.com/paper/2308.16529/) (2023)
- [Decision Making in Joint Push-Grasp Action Space for Large-Scale Object Sorting](https://mlsanity.com/paper/2010.10064/) (2020)
- [CycleIK: Neuro-inspired Inverse Kinematics](https://mlsanity.com/paper/2307.11554/) (2023)
- [Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2205.15862/) (2022)
- [Effect of Human Involvement on Work Performance and Fluency in Human-Robot Collaboration for Recycling](https://mlsanity.com/paper/2201.07990/) (2022)
- [CoBaIR: A Python Library for Context-Based Intention Recognition in Human-Robot-Interaction](https://mlsanity.com/paper/2308.05627/) (2023)

## Built on by (most cited first, 3 of 3)

- [Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation](https://mlsanity.com/paper/2407.13505/) (2024)
- [Pointing-Guided Target Estimation via Transformer-Based Attention](https://mlsanity.com/paper/2509.05031/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)

## Abstract (arXiv)

Human intention-based systems enable robots to perceive and interpret user actions to interact with humans and adapt to their behavior proactively. Therefore, intention prediction is pivotal in creating a natural interaction with social robots in human-designed environments. In this paper, we examine using Large Language Models (LLMs) to infer human intention in a collaborative object categorization task with a physical robot. We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions in a hierarchical architecture. Our evaluation of five LLMs shows the potential for reasoning about verbal and non-verbal user cues, leveraging their context-understanding and real-world knowledge to support intention prediction while collaborating on a task with a social robot. Video: https://youtu.be/tBJHfAuzohI