# ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

arXiv: https://arxiv.org/abs/2609.28955 · HTML: https://mlsanity.com/paper/2609.28955/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models.
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- Vision-language-action models / [Human-video and latent-action pretraining](https://mlsanity.com/vla/human-video-and-latent-action-pretraining/)

## Builds on (most cited first, 30 of 33)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Robotic Control via Embodied Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.08693/) (2024)
- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data](https://mlsanity.com/paper/2505.03233/) (2025)
- [ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver](https://mlsanity.com/paper/2508.10333/) (2025)
- [Mechanistic interpretability for steering vision-language-action models](https://mlsanity.com/paper/2509.00328/) (2025)
- [When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs](https://mlsanity.com/paper/2602.17659/) (2026)
- [AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention](https://mlsanity.com/paper/2511.18960/) (2025)
- [SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulation](https://mlsanity.com/paper/2511.10518/) (2025)
- [Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models](https://mlsanity.com/paper/2502.10090/) (2025)
- [VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing](https://mlsanity.com/paper/2605.30117/) (2026)
- [Making Foresight Actionable: Repurposing Representation Alignment in World Action Models](https://mlsanity.com/paper/2606.12217/) (2026)
- [Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding](https://mlsanity.com/paper/2512.22519/) (2025)
- [Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2603.23202/) (2026)
- [GazeVLA: Learning Human Intention for Robotic Manipulation](https://mlsanity.com/paper/2604.22615/) (2026)
- [Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation](https://mlsanity.com/paper/2605.30282/) (2026)
- [FocusVLA: Focused Visual Utilization for Vision-Language-Action Models](https://mlsanity.com/paper/2603.28740/) (2026)
- [Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models](https://mlsanity.com/paper/2510.16344/) (2025)
- [Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Models](https://mlsanity.com/paper/2607.10655/) (2026)
- [ShapeForce: Low-Cost Soft Robotic Wrist for Contact-Rich Manipulation](https://mlsanity.com/paper/2511.19955/) (2025)
- [IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance](https://mlsanity.com/paper/2601.16207/) (2026)
- [On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learning](https://mlsanity.com/paper/2605.26649/) (2026)
- [Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models](https://mlsanity.com/paper/2608.02197/) (2026)

## Abstract (arXiv)

Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.