mlsanityData to 7 Oct 2026

Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Tasks

arXiv 2205.09683 ↗First posted 19 May 2022cs.RO cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper describes a deep reinforcement learning (DRL) approach that won Phase 1 of the Real Robot Challenge (RRC) 2021, and then extends this method to a more difficult manipulation task. The RRC consisted of using a TriFinger robot to manipulate a cube along a specified positional trajectory, but with no requirement for the cube to have any specific orientation. We used a relatively simple reward function, a combination of goal-based sparse reward and distance reward, in conjunction with Hindsight Experience Replay (HER) to guide the learning of the DRL agent (Deep Deterministic Policy Gradient (DDPG)). Our approach allowed our agents to acquire dexterous robotic manipulation strategies in simulation. These strategies were then applied to the real robot and outperformed all other competition submissions, including those using more traditional robotic control techniques, in the final evaluation stage of the RRC. Here we extend this method, by modifying the task of Phase 1 of the RRC to require the robot to maintain the cube in a particular orientation, while the cube is moved along the required positional trajectory. The requirement to also orient the cube makes the agent unable to learn the task through blind exploration due to increased problem complexity. To circumvent this issue, we make novel use of a Knowledge Transfer (KT) technique that allows the strategies learned by the agent in the original task (which was agnostic to cube orientation) to be transferred to this task (where orientation matters). KT allowed the agent to learn and perform the extended task in the simulator, which improved the average positional deviation from 0.134 m to 0.02 m, and average orientation deviation from 142deg to 76deg during evaluation. This KT concept shows good generalisation properties and could be applied to any actor-critic learning algorithm.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

29 of 43 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2016OpenAI Gymcited by 1,684
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
17 more
  1. 2017Deep Reinforcement Learning that Matterscited by 727
  2. 2019Solving Rubik's Cube with a Robot Handcited by 699
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  7. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  8. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  9. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  10. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  11. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  12. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 34
  13. 2021Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFingercited by 34
  14. 2020TriFinger: An Open-Source Robot for Learning Dexteritycited by 31
  15. 2021Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulationcited by 14
  16. 2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8
  17. 2021Solving the Real Robot Challenge using Deep Reinforcement Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1.

5 papers.

  1. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  2. 2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
  3. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policiescited by 5
  4. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1
  5. 2023Sample-Efficient Reinforcement Learning with Symmetry-Guided Demonstrations for Robotic Manipulationcited by 0