# Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation

arXiv: https://arxiv.org/abs/2603.23202 · HTML: https://mlsanity.com/paper/2603.23202/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 28 of 28)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation](https://mlsanity.com/paper/2406.04339/) (2024)
- [SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation](https://mlsanity.com/paper/2501.18564/) (2025)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12894/) (2024)
- [Integrating Human Gaze into Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/2011.03920/) (2020)
- [AGIL: Learning Attention from Human for Visuomotor Tasks](https://mlsanity.com/paper/1806.03960/) (2018)
- [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](https://mlsanity.com/paper/2401.09454/) (2023)
- [VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models](https://mlsanity.com/paper/2402.01735/) (2024)
- [Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation](https://mlsanity.com/paper/2102.01295/) (2021)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [VIP: Vision Instructed Pre-training for Robotic Manipulation](https://mlsanity.com/paper/2410.07169/) (2024)
- [Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://mlsanity.com/paper/2507.15833/) (2025)
- [Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding](https://mlsanity.com/paper/2510.21356/) (2025)
- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [Selective Eye-gaze Augmentation To Enhance Imitation Learning In Atari Games](https://mlsanity.com/paper/2012.03145/) (2020)
- [Multi-task real-robot data with gaze attention for dual-arm fine manipulation](https://mlsanity.com/paper/2401.07603/) (2024)
- [From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2504.11368/) (2025)
- [Gaze-Guided Learning: Avoiding Shortcut Bias in Visual Classification](https://mlsanity.com/paper/2504.05583/) (2025)
- [Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation](https://mlsanity.com/paper/2501.15071/) (2025)
- [Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze Information and Motion Bottlenecks](https://mlsanity.com/paper/2502.18121/) (2025)
- [MindEye-OmniAssist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution](https://mlsanity.com/paper/2503.13250/) (2025)

## Built on by (most cited first, 9 of 9)

- [ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI](https://mlsanity.com/paper/2607.04426/) (2026)
- [Inference-Time Attention Steering for Vision-Language-Action Driving Models](https://mlsanity.com/paper/2608.17095/) (2026)
- [Vision-Language-Action Models: Experimental Insights from a Real-World UR5 Platform](https://mlsanity.com/paper/2606.30456/) (2026)
- [Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models](https://mlsanity.com/paper/2608.23478/) (2026)
- [Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation](https://mlsanity.com/paper/2609.05369/) (2026)
- [ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation](https://mlsanity.com/paper/2609.28955/) (2026)
- [Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning](https://mlsanity.com/paper/2609.34550/) (2026)
- [EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras](https://mlsanity.com/paper/2610.03710/) (2026)
- [DIVA: Dual-Space Intent-Aware Visual Attenuation for Vision-Language-Action Policies](https://mlsanity.com/paper/2610.09144/) (2026)

## Abstract (arXiv)

Despite advances in Vision-Language-Action (VLA) models, robotic manipulation struggles with fine-grained tasks because current models lack mechanisms for active visual attention allocation. Human gaze naturally encodes intent, planning, and execution patterns -- offering a powerful supervisory signal for guiding robot perception. We introduce a gaze-regularized training framework that aligns VLA models' internal attention with human visual patterns without architectural modifications or inference-time overhead. Our method transforms temporally aggregated gaze heatmaps into patch-level distributions and regularizes the transformer's attention through KL divergence, creating an inductive bias toward task-relevant features while preserving deployment efficiency. When integrated into existing VLA architectures, our approach yields 4-12% improvements across manipulation benchmarks. The gaze-regularized models reach equivalent performance with fewer training steps and maintain robustness under lighting variations and sensor noise. Beyond performance metrics, the learned attention patterns produce interpretable visualizations that mirror human strategies, enhancing trust in robotic systems. Moreover, our framework requires no eye-tracking equipment and applies directly to existing datasets. These results demonstrate that human perceptual priors can significantly accelerate robot learning while improving both task performance and system interpretability.