mlsanityData to 7 Oct 2026

Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection

arXiv 1603.02199 ↗First posted 7 Mar 2016cs.LG cs.AI cs.CVcited by 312

Not on the map. Why?

Cited by 43 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing.

Key facts. Cited by 312 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 3; 2021: 75; 2022: 73; 2023: 55; 2024: 52; 2025: 26; 2026: 25.

312 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  5. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  6. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  7. 2023Tool Learning with Foundation Modelscited by 302
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  9. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  10. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  11. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  12. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
38 more
  1. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  2. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  3. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  4. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  5. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  6. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  7. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  8. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  9. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  10. 2023On Bringing Robots Homecited by 101
  11. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  12. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  13. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  14. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  15. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  16. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  17. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  18. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  19. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  20. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  21. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  22. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 68
  23. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  24. 2021GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detectioncited by 66
  25. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  26. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  27. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 59
  28. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  29. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 56
  30. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  31. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  32. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  33. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  34. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  35. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  36. 2021O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learningcited by 46
  37. 2021Auto-Tuned Sim-to-Real Transfercited by 45
  38. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45

50 most cited of 312 All 312 (JSON)