mlsanityData to 7 Oct 2026

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

arXiv 2609.28955 ↗First posted 24 Sep 2026cs.ROcited by 0

On the map: Vision-language-action models → Human-video and latent-action pretraining also fits: Failure recovery and robustness

Abstract (arXiv)

Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models. Builds on 33 of its 40 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

33 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data6 May

Other references · 32

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  7. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  8. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  9. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  10. 2017Visualizing and Understanding Atari Agentscited by 141
  11. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 80
  12. 2025Mechanistic interpretability for steering vision-language-action modelscited by 30
20 more
  1. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 27
  2. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  3. 2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 20
  4. 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 13
  5. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Groundingcited by 10
  6. 2026VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracingcited by 9
  7. 2026Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulationcited by 8
  8. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 8
  9. 2026Gaze-Regularized Vision-Language-Action Models for Robotic Manipulationcited by 7
  10. 2026GazeVLA: Learning Human Intention for Robotic Manipulationcited by 7
  11. 2025Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Modelscited by 5
  12. 2026FocusVLA: Focused Visual Utilization for Vision-Language-Action Modelscited by 5
  13. 2026Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Modelscited by 4
  14. 2025ShapeForce: Low-Cost Soft Robotic Wrist for Contact-Rich Manipulationcited by 2
  15. 2026On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learningcited by 2
  16. 2026CofactVLA: Deconfounding Vision-Language-Action Models via Counterfactual Interventioncited by 2
  17. 2026IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidancecited by 1
  18. 2026GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learningcited by 1
  19. 2026Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Modelscited by 1
  20. 2026Pointing-VLA: Typed Spatial Grounding Interfaces for Vision-Language-Action Manipulationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning
  2. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  3. 2026Uncovering and Mitigating Positional Blind Spots in Vision-Language-Action Models
  4. 2026ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistency
  5. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies
  6. 2026Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models
  7. 2025PosA-VLA: Enhancing Action Generation via Pose-Conditioned Anchor Attention
  8. 2026Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation