# IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models

arXiv: https://arxiv.org/abs/2604.24002 · HTML: https://mlsanity.com/paper/2604.24002/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020)
- [COMBO: Compositional World Models for Embodied Multi-Agent Cooperation](https://mlsanity.com/paper/2404.10775/) (2024)
- [Foundation Models for Video Understanding: A Survey](https://mlsanity.com/paper/2405.03770/) (2024)
- [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025)
- [Cognitive science as a source of forward and inverse models of human decisions for robotics and control](https://mlsanity.com/paper/2109.00127/) (2021)
- [GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment](https://mlsanity.com/paper/2403.11075/) (2024)
- [I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models](https://mlsanity.com/paper/2509.16072/) (2025)
- [Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction](https://mlsanity.com/paper/1907.02426/) (2019)
- [Infer Human's Intentions Before Following Natural Language Instructions](https://mlsanity.com/paper/2409.18073/) (2024)
- [Foundation Model Driven Robotics: A Comprehensive Review](https://mlsanity.com/paper/2507.10087/) (2025)
- [INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning](https://mlsanity.com/paper/2412.03565/) (2024)
- [USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions](https://mlsanity.com/paper/2502.10636/) (2025)
- [Deep Learning Approaches for Multimodal Intent Recognition: A Survey](https://mlsanity.com/paper/2507.22934/) (2025)
- [Goal Recognition using Actor-Critic Optimization](https://mlsanity.com/paper/2501.01463/) (2024)
- [Reasoning LLMs for User-Aware Multimodal Conversational Agents](https://mlsanity.com/paper/2504.01700/) (2025)
- [Two ways to make your robot proactive: reasoning about human intentions, or reasoning about possible futures](https://mlsanity.com/paper/2205.05492/) (2022)
- [LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application](https://mlsanity.com/paper/2406.13787/) (2024)
- [Demographic User Modeling for Social Robotics with Multimodal Pre-trained Models](https://mlsanity.com/paper/2502.10642/) (2025)
- [Proceedings of 1st Workshop on Advancing Artificial Intelligence through Theory of Mind](https://mlsanity.com/paper/2505.03770/) (2025)
- [HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs](https://mlsanity.com/paper/2601.19839/) (2026)

## Abstract (arXiv)

Improving the effectiveness of human-robot interaction requires social robots to accurately infer human goals through robust intention understanding. This challenge is particularly critical in multimodal settings, where agents must integrate heterogeneous signals including text, visual cues to form a coherent interpretation of user intent. This paper presents IntentVLM, a novel two-stage video-language framework designed for open-vocabulary human intention recognition. The approach is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation followed by structured inference through selection, effectively reducing hallucinations in latent reasoning. Evaluated on the IntentQA and Inst-IT Bench datasets, IntentVLM achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance. Our findings demonstrate that this structured reasoning approach enhances open-vocabulary intention understanding without catastrophic forgetting, offering a robust foundation for human-centered robotics.