# What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

arXiv: https://arxiv.org/abs/2609.05376 · HTML: https://mlsanity.com/paper/2609.05376/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 12 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation](https://mlsanity.com/paper/2307.03659/) (2023)
- [RoboGround: Robotic Manipulation with Grounded Vision-Language Priors](https://mlsanity.com/paper/2504.21530/) (2025)
- [Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robust](https://mlsanity.com/paper/2410.01971/) (2024)
- [GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization](https://mlsanity.com/paper/2605.12369/) (2026)
- [Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT](https://mlsanity.com/paper/2508.08748/) (2025)
- [Improving Generalization Ability of Robotic Imitation Learning by Resolving Causal Confusion in Observations](https://mlsanity.com/paper/2507.22380/) (2025)
- [ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy](https://mlsanity.com/paper/2502.09649/) (2025)
- [Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulation](https://mlsanity.com/paper/2603.04845/) (2026)

## Abstract (arXiv)

Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.