# Economical Precise Manipulation and Auto Eye-Hand Coordination with Binocular Visual Reinforcement Learning

arXiv: https://arxiv.org/abs/2205.05963 · HTML: https://mlsanity.com/paper/2205.05963/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Deep Reinforcement Learning for High Precision Assembly Tasks](https://mlsanity.com/paper/1708.04033/) (2017)
- [A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning](https://mlsanity.com/paper/1810.01531/) (2018)
- [Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks](https://mlsanity.com/paper/2004.14404/) (2020)
- [Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2008.10224/) (2020)
- [Learning Robotic Assembly from CAD](https://mlsanity.com/paper/1803.07635/) (2018)
- [Quickly Inserting Pegs into Uncertain Holes using Multi-view Images and Deep Network Trained on Synthetic Data](https://mlsanity.com/paper/1902.09157/) (2019)
- ["Good Robot!": Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer](https://mlsanity.com/paper/1909.11730/) (2019)

## Abstract (arXiv)

Precision robotic manipulation tasks (insertion, screwing, precisely pick, precisely place) are required in many scenarios. Previous methods achieved good performance on such manipulation tasks. However, such methods typically require tedious calibration or expensive sensors. 3D/RGB-D cameras and torque/force sensors add to the cost of the robotic application and may not always be economical. In this work, we aim to solve these but using only weak-calibrated and low-cost webcams. We propose Binocular Alignment Learning (BAL), which could automatically learn the eye-hand coordination and points alignment capabilities to solve the four tasks. Our work focuses on working with unknown eye-hand coordination and proposes different ways of performing eye-in-hand camera calibration automatically. The algorithm was trained in simulation and used a practical pipeline to achieve sim2real and test it on the real robot. Our method achieves a competitively good result with minimal cost on the four tasks.