# Pay attention! - Robustifying a Deep Visuomotor Policy through Task-Focused Attention

arXiv: https://arxiv.org/abs/1809.10093 · HTML: https://mlsanity.com/paper/1809.10093/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2018.

## Built on by (most cited first, 11 of 11)

- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [Certifiably Robust Interpretation via Renyi Differential Privacy](https://mlsanity.com/paper/2107.01561/) (2021)
- [Mobile Robot Manipulation using Pure Object Detection](https://mlsanity.com/paper/2201.12437/) (2022)
- [C3DM: Constrained-Context Conditional Diffusion Models for Imitation Learning](https://mlsanity.com/paper/2311.01419/) (2023)
- [Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping](https://mlsanity.com/paper/2104.00878/) (2021)
- [Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints](https://mlsanity.com/paper/2409.11518/) (2024)
- [MoVEInt: Mixture of Variational Experts for Learning Human-Robot Interactions from Demonstrations](https://mlsanity.com/paper/2407.07636/) (2024)
- [Video Generation from Text Employing Latent Path Construction for Temporal Modeling](https://mlsanity.com/paper/2107.13766/) (2021)
- [See Less, Specify More: Visual Evidence Budgets for Generalizable VLAs](https://mlsanity.com/paper/2606.02735/) (2026)

## Abstract (arXiv)

Several recent studies have demonstrated the promise of deep visuomotor policies for robot manipulator control. Despite impressive progress, these systems are known to be vulnerable to physical disturbances, such as accidental or adversarial bumps that make them drop the manipulated object. They also tend to be distracted by visual disturbances such as objects moving in the robot's field of view, even if the disturbance does not physically prevent the execution of the task. In this paper, we propose an approach for augmenting a deep visuomotor policy trained through demonstrations with Task Focused visual Attention (TFA). The manipulation task is specified with a natural language text such as `move the red bowl to the left'. This allows the visual attention component to concentrate on the current object that the robot needs to manipulate. We show that even in benign environments, the TFA allows the policy to consistently outperform a variant with no attention mechanism. More importantly, the new policy is significantly more robust: it regularly recovers from severe physical disturbances (such as bumps causing it to drop the object) from which the baseline policy, i.e. with no visual attention, almost never recovers. In addition, we show that the proposed policy performs correctly in the presence of a wide class of visual disturbances, exhibiting a behavior reminiscent of human selective visual attention experiments. Our proposed approach consists of a VAE-GAN network which encodes the visual input and feeds it to a Motor network that moves the robot joints. Also, our approach benefits from a teacher network for the TFA that leverages textual input command to robustify the visual encoder against various types of disturbances.