# End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation

arXiv: https://arxiv.org/abs/2202.06027 · HTML: https://mlsanity.com/paper/2202.06027/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 12 Feb 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Adversarial Examples Improve Image Recognition](https://mlsanity.com/paper/1911.09665/) (2019)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Reinforcement and Imitation Learning for Diverse Visuomotor Skills](https://mlsanity.com/paper/1802.09564/) (2018)
- [Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations](https://mlsanity.com/paper/2003.06085/) (2020)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [Explainability in Deep Reinforcement Learning](https://mlsanity.com/paper/2008.06693/) (2020)
- [Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1907.13098/) (2019)
- [KETO: Learning Keypoint Representations for Tool Manipulation](https://mlsanity.com/paper/1910.11977/) (2019)
- [Viewpoints and Keypoints](https://mlsanity.com/paper/1411.6067/) (2014)
- [Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control](https://mlsanity.com/paper/1511.03791/) (2015)
- [Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods](https://mlsanity.com/paper/1802.10264/) (2018)
- [Robotic Grasping using Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04499/) (2020)
- [kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion](https://mlsanity.com/paper/1909.06980/) (2019)
- [Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics](https://mlsanity.com/paper/1901.08651/) (2019)
- [KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation](https://mlsanity.com/paper/2007.13960/) (2020)
- [6D Pose Estimation with Correlation Fusion](https://mlsanity.com/paper/1909.12936/) (2019)
- [Efficient Robotic Task Generalization Using Deep Model Fusion Reinforcement Learning](https://mlsanity.com/paper/1912.05205/) (2019)

## Built on by (most cited first, 2 of 2)

- [Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learning](https://mlsanity.com/paper/2403.17266/) (2024)
- [Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)](https://mlsanity.com/paper/2312.00592/) (2023)

## Abstract (arXiv)

We present an end-to-end Reinforcement Learning(RL) framework for robotic manipulation tasks, using a robust and efficient keypoints representation. The proposed method learns keypoints from camera images as the state representation, through a self-supervised autoencoder architecture. The keypoints encode the geometric information, as well as the relationship of the tool and target in a compact representation to ensure efficient and robust learning. After keypoints learning, the RL step then learns the robot motion from the extracted keypoints state representation. The keypoints and RL learning processes are entirely done in the simulated environment. We demonstrate the effectiveness of the proposed method on robotic manipulation tasks including grasping and pushing, in different scenarios. We also investigate the generalization capability of the trained model. In addition to the robust keypoints representation, we further apply domain randomization and adversarial training examples to achieve zero-shot sim-to-real transfer in real-world robotic manipulation tasks.