What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
Not on the map. Why?
Abstract (arXiv)
Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 12 references that are in the map. First posted on arXiv 4 Sep 2026.
Builds on
12 of 12 references.
References · 12
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
- 2019Causal Confusion in Imitation Learningcited by 288
- 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 82
- 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
- 2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
- 2026GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specializationcited by 10
- 2025Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACTcited by 5
- 2025Improving Generalization Ability of Robotic Imitation Learning by Resolving Causal Confusion in Observationscited by 4
- 2025ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policycited by 1
- 2026Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera
- 2026GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning
- 2026More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning
- 2021Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery
- 2026Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking
- 2025DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization
- 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control
- 2025A Study on Enhancing the Generalization Ability of Visuomotor Policies via Data Augmentation ↗