mlsanityData to 7 Oct 2026

Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control

arXiv 2103.00375 ↗First posted 28 Feb 2021cs.RO cs.AI cs.LGcited by 24

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation Learning (IL) is an effective framework to learn visuomotor skills from offline demonstration data. However, IL methods often fail to generalize to new scene configurations not covered by training data. On the other hand, humans can manipulate objects in varying conditions. Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location. In this work, we present a learnable action space, Hand-eye Action Networks (HAN), that can approximate human's hand-eye coordination behaviors by learning from human teleoperated demonstrations. Through a set of challenging multi-stage manipulation tasks, we show that a visuomotor policy equipped with HAN is able to inherit the key spatial invariance property of hand-eye coordination and achieve zero-shot generalization to new scene configurations. Additional materials available at https://sites.google.com/stanford.edu/han

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 51 references that are in the map. First posted on arXiv 28 Feb 2021.

Builds on

27 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  2. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep
  3. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov

Other references · 24

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  4. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  5. 2014Recurrent Models of Visual Attentioncited by 344
  6. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  7. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  8. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  9. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  10. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  11. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  12. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
12 more
  1. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  2. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  3. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  4. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  5. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  6. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 68
  7. 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
  8. 2020Neuroevolution of Self-Interpretable Agentscited by 48
  9. 2020Spatial Action Maps for Mobile Manipulationcited by 31
  10. 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17
  11. 2019Reinforcement Learning with Attention that Works: A Self-Supervised Approachcited by 17
  12. 2020Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulationcited by 17

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 9; 2024: 7; 2025: 5.

24 papers.

Sort:

By area:

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  2. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  3. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  4. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  5. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  6. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  7. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  8. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  9. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  10. 2023Waypoint-Based Imitation Learning for Robotic Manipulationcited by 59
  11. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  12. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
12 more
  1. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  2. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  3. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  4. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  5. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  6. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 18
  7. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  8. 2023C3DM: Constrained-Context Conditional Diffusion Models for Imitation Learningcited by 7
  9. 2025Toward Embodied AGI: A Review of Embodied AI and the Road Aheadcited by 7
  10. 2023Generalizable Imitation Learning Through Pre-Trained Representationscited by 3
  11. 2025Learning Predictive Visuomotor Coordinationcited by 2
  12. 2025Learning human-to-robot handovers through 3D scene reconstructioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025SViP: Sequencing Bimanual Visuomotor Policies with Object-Centric Motion Primitives
  2. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  3. 2026Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations ↗
  4. 2026Learning Coordinated Visuomotor Box-Pushing from Solo Demonstrations
  5. 2025A Study on Enhancing the Generalization Ability of Visuomotor Policies via Data Augmentation ↗
  6. 2021Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
  7. 2026Task Editing for Generalizable 3D Visuomotor Policy Learning
  8. 2025Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation