# Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation arXiv: https://arxiv.org/abs/2605.30282 · HTML: https://mlsanity.com/paper/2605.30282/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. - Builds on 30 of its 35 references that are in the map. - First posted on arXiv 28 May 2026. ## Placed in - Vision-language-action models / [Failure recovery and robustness](https://mlsanity.com/vla/failure-recovery-and-robustness/) ## Builds on (most cited first, 30 of 30) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025) - [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025) - [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023) - [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023) - [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021) - [Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models](https://mlsanity.com/paper/2501.14818/) (2025) - [Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding](https://mlsanity.com/paper/2507.00416/) (2025) - [RoboGround: Robotic Manipulation with Grounded Vision-Language Priors](https://mlsanity.com/paper/2504.21530/) (2025) - [ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models](https://mlsanity.com/paper/2506.16211/) (2025) - [Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignment](https://mlsanity.com/paper/2511.04555/) (2025) - [Learning Precise Affordances from Egocentric Videos for Robotic Manipulation](https://mlsanity.com/paper/2408.10123/) (2024) - [PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2601.07060/) (2026) - [Gaze-based, Context-aware Robotic System for Assisted Reaching and Grasping](https://mlsanity.com/paper/1809.08095/) (2018) - [VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models](https://mlsanity.com/paper/2603.22003/) (2026) - [Evo-Depth: A Lightweight Depth-Enhanced Vision-Language-Action Model](https://mlsanity.com/paper/2605.14950/) (2026) - [Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding](https://mlsanity.com/paper/2510.21356/) (2025) - [Point What You Mean: Visually Grounded Instruction Policy](https://mlsanity.com/paper/2512.18933/) (2025) - [GazeVLA: Learning Human Intention for Robotic Manipulation](https://mlsanity.com/paper/2604.22615/) (2026) - [REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?](https://mlsanity.com/paper/2505.10872/) (2025) - [Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models](https://mlsanity.com/paper/2601.05336/) (2026) ## Built on by (most cited first, 8 of 8) - [LA4VLA: Learning to Act without Seeing via Language-Action Pretraining](https://mlsanity.com/paper/2606.27295/) (2026) - [GIVE: Grounding Human Gestures in Vision-Language-Action Models](https://mlsanity.com/paper/2606.13435/) (2026) - [mmHRI: Towards Privacy-Preserving Human-Robot Interaction with Millimeter-Wave Radar](https://mlsanity.com/paper/2609.34220/) (2026) - [SpaceVLA: Spatially Grounded VLA for Robotic Manipulation with User-Authored Grasp and Place Anchors](https://mlsanity.com/paper/2608.05730/) (2026) - [Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models](https://mlsanity.com/paper/2608.23478/) (2026) - [DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA](https://mlsanity.com/paper/2608.28108/) (2026) - [ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation](https://mlsanity.com/paper/2609.28955/) (2026) - [Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors](https://mlsanity.com/paper/2610.01794/) (2026) ## Abstract (arXiv) Vision-Language-Action (VLA) models have recently shown strong potential for robot learning by following language instructions. However, in practice, language alone is often insufficient to precisely convey human intent. It is difficult to describe which exact object to interact with among similar candidates, where to act on the object, or how the target may change during execution. To address this limitation, we propose Gaze2Act, a novel VLA framework that leverages human gaze as a dynamic and intuitive intent signal for complex interactive manipulation. Gaze2Act first bridges the ego-exo view gap by mapping first-person gaze into the robot's perspective through cross-view semantic matching, producing both an object mask and a gaze point for coarse-to-fine target specification. These cues are then integrated into the policy through perception-level prompting and action-level conditioning, allowing the robot to attend to relevant regions and execute precise interactions under dynamic intent. In a systematic evaluation across seven task categories and 16 real-robot tasks on a Unitree G1 humanoid, Gaze2Act achieves state-of-the-art performance in both intent accuracy and task success rate. It notably outperforms baselines in object disambiguation, fine-grained interaction, and dynamic intent steering. These results demonstrate that human gaze provides a natural, low-burden, and highly expressive modality for human-in-the-loop VLA control.