# NVP-HRI: Zero Shot Natural Voice and Posture-based Human-Robot Interaction via Large Language Model

arXiv: https://arxiv.org/abs/2503.09335 · HTML: https://mlsanity.com/paper/2503.09335/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 109 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning](https://mlsanity.com/paper/1802.08705/) (2018)
- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing](https://mlsanity.com/paper/2305.10345/) (2023)
- [Chat with the Environment: Interactive Multimodal Perception Using Large Language Models](https://mlsanity.com/paper/2303.08268/) (2023)
- [Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2303.10457/) (2023)
- [MMAUD: A Comprehensive Multi-Modal Anti-UAV Dataset for Modern Miniature Drone Threats](https://mlsanity.com/paper/2402.03706/) (2024)
- [MoPA: Multi-Modal Prior Aided Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2309.11839/) (2023)
- [GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning](https://mlsanity.com/paper/2208.14326/) (2022)
- [Segregator: Global Point Cloud Registration with Semantic and Geometric Cues](https://mlsanity.com/paper/2301.07425/) (2023)
- [Co-Speech Gesture Synthesis using Discrete Gesture Token Learning](https://mlsanity.com/paper/2303.12822/) (2023)
- [Outram: One-shot Global Localization via Triangulated Scene Graph and Global Outlier Pruning](https://mlsanity.com/paper/2309.08914/) (2023)
- [HCTO: Optimality-Aware LiDAR Inertial Odometry with Hybrid Continuous Time Optimization for Compact Wearable Mapping System](https://mlsanity.com/paper/2403.14173/) (2024)
- [DIRECT: A Differential Dynamic Programming Based Framework for Trajectory Generation](https://mlsanity.com/paper/2109.04686/) (2021)
- [A Cost-Effective Cooperative Exploration and Inspection Strategy for Heterogeneous Aerial System](https://mlsanity.com/paper/2403.01225/) (2024)
- [AIR-Embodied: An Efficient Active 3DGS-based Interaction and Reconstruction Framework with Embodied Large Language Model](https://mlsanity.com/paper/2409.16019/) (2024)
- [Robust Loop Closure by Textual Cues in Challenging Environments](https://mlsanity.com/paper/2410.15869/) (2024)
- [AV-PedAware: Self-Supervised Audio-Visual Fusion for Dynamic Pedestrian Awareness](https://mlsanity.com/paper/2411.06789/) (2024)

## Built on by (most cited first, 1 of 1)

- [GIVE: Grounding Human Gestures in Vision-Language-Action Models](https://mlsanity.com/paper/2606.13435/) (2026)

## Abstract (arXiv)

Effective Human-Robot Interaction (HRI) is crucial for future service robots in aging societies. Existing solutions are biased toward only well-trained objects, creating a gap when dealing with new objects. Currently, HRI systems using predefined gestures or language tokens for pretrained objects pose challenges for all individuals, especially elderly ones. These challenges include difficulties in recalling commands, memorizing hand gestures, and learning new names. This paper introduces NVP-HRI, an intuitive multi-modal HRI paradigm that combines voice commands and deictic posture. NVP-HRI utilizes the Segment Anything Model (SAM) to analyze visual cues and depth data, enabling precise structural object representation. Through a pre-trained SAM network, NVP-HRI allows interaction with new objects via zero-shot prediction, even without prior knowledge. NVP-HRI also integrates with a large language model (LLM) for multimodal commands, coordinating them with object selection and scene distribution in real time for collision-free trajectory solutions. We also regulate the action sequence with the essential control syntax to reduce LLM hallucination risks. The evaluation of diverse real-world tasks using a Universal Robot showcased up to 59.2% efficiency improvement over traditional gesture control, as illustrated in the video https://youtu.be/EbC7al2wiAc. Our code and design will be openly available at https://github.com/laiyuzhi/NVP-HRI.git.