# Solving the Real Robot Challenge using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2109.15233 · HTML: https://mlsanity.com/paper/2109.15233/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers](https://mlsanity.com/paper/2006.13916/) (2020)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review](https://mlsanity.com/paper/2102.04148/) (2021)
- [TriFinger: An Open-Source Robot for Learning Dexterity](https://mlsanity.com/paper/2008.03596/) (2020)
- [Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation](https://mlsanity.com/paper/2105.02087/) (2021)
- [Grasp and Motion Planning for Dexterous Manipulation for the Real Robot Challenge](https://mlsanity.com/paper/2101.02842/) (2021)

## Built on by (most cited first, 5 of 5)

- [Improving Behavioural Cloning with Positive Unlabeled Learning](https://mlsanity.com/paper/2301.11734/) (2023)
- [Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research](https://mlsanity.com/paper/2204.10297/) (2022)
- [Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Tasks](https://mlsanity.com/paper/2205.09683/) (2022)
- [Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies](https://mlsanity.com/paper/2301.13019/) (2023)
- [Dataset Clustering for Improved Offline Policy Learning](https://mlsanity.com/paper/2402.09550/) (2024)

## Abstract (arXiv)

This paper details our winning submission to Phase 1 of the 2021 Real Robot Challenge; a challenge in which a three-fingered robot must carry a cube along specified goal trajectories. To solve Phase 1, we use a pure reinforcement learning approach which requires minimal expert knowledge of the robotic system, or of robotic grasping in general. A sparse, goal-based reward is employed in conjunction with Hindsight Experience Replay to teach the control policy to move the cube to the desired x and y coordinates of the goal. Simultaneously, a dense distance-based reward is employed to teach the policy to lift the cube to the z coordinate (the height component) of the goal. The policy is trained in simulation with domain randomisation before being transferred to the real robot for evaluation. Although performance tends to worsen after this transfer, our best policy can successfully lift the real cube along goal trajectories via an effective pinching grasp. Our approach outperforms all other submissions, including those leveraging more traditional robotic control techniques, and is the first pure learning-based method to solve this challenge.