# Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning

arXiv: https://arxiv.org/abs/2609.34550 · HTML: https://mlsanity.com/paper/2609.34550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Vision-language-action models / [Failure recovery and robustness](https://mlsanity.com/vla/failure-recovery-and-robustness/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)
- [ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning](https://mlsanity.com/paper/2507.16815/) (2025)
- [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025)
- [ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver](https://mlsanity.com/paper/2508.10333/) (2025)
- [RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation](https://mlsanity.com/paper/2411.02704/) (2024)
- [AGIL: Learning Attention from Human for Visuomotor Tasks](https://mlsanity.com/paper/1806.03960/) (2018)
- [LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries](https://mlsanity.com/paper/2601.15197/) (2026)
- [Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop](https://mlsanity.com/paper/2506.10968/) (2025)
- [AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention](https://mlsanity.com/paper/2511.18960/) (2025)
- [Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy](https://mlsanity.com/paper/2512.11218/) (2025)
- [VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models](https://mlsanity.com/paper/2603.22003/) (2026)
- [Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models](https://mlsanity.com/paper/2603.19233/) (2026)
- [Stable Language Guidance for Vision-Language-Action Models](https://mlsanity.com/paper/2601.04052/) (2026)
- [Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://mlsanity.com/paper/2507.15833/) (2025)
- [AimBot: A Simple Auxiliary Visual Cue to Enhance Spatial Awareness of Visuomotor Policies](https://mlsanity.com/paper/2508.08113/) (2025)
- [Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2603.23202/) (2026)
- [Selective Eye-gaze Augmentation To Enhance Imitation Learning In Atari Games](https://mlsanity.com/paper/2012.03145/) (2020)
- [FocusVLA: Focused Visual Utilization for Vision-Language-Action Models](https://mlsanity.com/paper/2603.28740/) (2026)
- [Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation](https://mlsanity.com/paper/2501.15071/) (2025)
- [TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models](https://mlsanity.com/paper/2603.24584/) (2026)
- [Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment](https://mlsanity.com/paper/2604.05614/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce eye-tracker-supervised gaze prompting, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with π0, gaze prompting increases mean success from 26.3% to 56.0% across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release textscGazeMani, a dataset of 1,200 teleoperated trajectories with synchronized gaze.