mlsanityData to 7 Oct 2026

Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning

arXiv 2609.34550 ↗First posted 28 Sep 2026cs.ROcited by 0

On the map: Vision-language-action models → Failure recovery and robustness also fits: Reasoning and chain-of-thought in VLA

Abstract (arXiv)

Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce eye-tracker-supervised gaze prompting, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with π0, gaze prompting increases mean success from 26.3% to 56.0% across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release textscGazeMani, a dataset of 1,200 teleoperated trajectories with synchronized gaze.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 29 of its 35 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

29 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models25 Mar
  2. 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries21 Jan
  3. 2026Stable Language Guidance for Vision-Language-Action Models7 Jan

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  7. 2023Vision Transformers Need Registerscited by 844
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  10. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  11. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  12. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 80
14 more
  1. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  2. 2018AGIL: Learning Attention from Human for Visuomotor Taskscited by 32
  3. 2025Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loopcited by 23
  4. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  5. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  6. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Modelscited by 17
  7. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Modelscited by 14
  8. 2025AimBot: A Simple Auxiliary Visual Cue to Enhance Spatial Awareness of Visuomotor Policiescited by 10
  9. 2025Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformerscited by 8
  10. 2026Gaze-Regularized Vision-Language-Action Models for Robotic Manipulationcited by 7
  11. 2020Selective Eye-gaze Augmentation To Enhance Imitation Learning In Atari Gamescited by 6
  12. 2026FocusVLA: Focused Visual Utilization for Vision-Language-Action Modelscited by 5
  13. 2025Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulationcited by 3
  14. 2026Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation
  2. 2026ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation
  3. 2026Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation
  4. 2026VISTA: Enhancing Visual Conditioning via Track-Following Preference Optimization in Vision-Language-Action Models
  5. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
  6. 2026Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models
  7. 2025MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulation
  8. 2026DTP: A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Models