# GIVE: Grounding Human Gestures in Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2606.13435 · HTML: https://mlsanity.com/paper/2606.13435/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding](https://mlsanity.com/paper/2507.00416/) (2025)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignment](https://mlsanity.com/paper/2511.04555/) (2025)
- [Learning Precise Affordances from Egocentric Videos for Robotic Manipulation](https://mlsanity.com/paper/2408.10123/) (2024)
- [RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Model](https://mlsanity.com/paper/2409.19590/) (2024)
- [PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2601.07060/) (2026)
- [Gesture-Informed Robot Assistance via Foundation Models](https://mlsanity.com/paper/2309.02721/) (2023)
- [Evo-Depth: A Lightweight Depth-Enhanced Vision-Language-Action Model](https://mlsanity.com/paper/2605.14950/) (2026)
- [Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation](https://mlsanity.com/paper/2605.30282/) (2026)
- [Diver Interest via Pointing: Human-Directed Object Inspection for AUVs](https://mlsanity.com/paper/2212.01205/) (2022)
- [GestLLM: Advanced Hand Gesture Interpretation via Large Language Models for Human-Robot Interaction](https://mlsanity.com/paper/2501.07295/) (2025)
- [IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interaction](https://mlsanity.com/paper/2510.07778/) (2025)
- [NVP-HRI: Zero Shot Natural Voice and Posture-based Human-Robot Interaction via Large Language Model](https://mlsanity.com/paper/2503.09335/) (2025)

## Built on by (most cited first, 1 of 1)

- [mmHRI: Towards Privacy-Preserving Human-Robot Interaction with Millimeter-Wave Radar](https://mlsanity.com/paper/2609.34220/) (2026)

## Abstract (arXiv)

Human communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.