mlsanityData to 7 Oct 2026

Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger

arXiv 2108.09779 ↗First posted 22 Aug 2021cs.RO cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

We present a system for learning a challenging dexterous manipulation task involving moving a cube to an arbitrary 6-DoF pose with only 3-fingers trained with NVIDIA's IsaacGym simulator. We show empirical benefits, both in simulation and sim-to-real transfer, of using keypoints as opposed to position+quaternion representations for the object pose in 6-DoF for policy observations and in reward calculation to train a model-free reinforcement learning agent. By utilizing domain randomization strategies along with the keypoint representation of the pose of the manipulated object, we achieve a high success rate of 83% on a remote TriFinger system maintained by the organizers of the Real Robot Challenge. With the aim of assisting further research in learning in-hand manipulation, we make the codebase of our system, along with trained checkpoints that come with billions of steps of experience available, at https://s2r2-ig.github.io

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 22 Aug 2021.

Builds on

21 of 38 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  8. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  10. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  11. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 237
  12. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
9 more
  1. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  2. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  3. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 78
  4. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77
  5. 2020TriFinger: An Open-Source Robot for Learning Dexteritycited by 31
  6. 2021Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulationcited by 14
  7. 2020Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbscited by 13
  8. 2021Grasp and Motion Planning for Dexterous Manipulation for the Real Robot Challengecited by 4
  9. 2021Dexterous Manipulation Primitives for the Real Robot Challengecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 9; 2024: 10; 2025: 4; 2026: 1.

34 papers.

Sort:

By area:

  1. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  2. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  3. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  4. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  5. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  6. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  7. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 92
  8. 2022Factory: Fast Contact for Robotic Assemblycited by 49
  9. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 24
  10. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  11. 2024Local Policies Enable Zero-shot Long-horizon Manipulationcited by 21
  12. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 20
22 more
  1. 2022Compositional Multi-Object Reinforcement Learning with Linear Relation Networkscited by 19
  2. 2024Symmetry Considerations for Learning Task Symmetric Robot Policiescited by 18
  3. 2024AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touchcited by 18
  4. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 12
  5. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  6. 2024CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objectscited by 8
  7. 2024Vegetable Peeling: A Case Study in Constrained Dexterous Manipulationcited by 7
  8. 2024DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robotscited by 7
  9. 2022Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Researchcited by 6
  10. 2022Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Taskscited by 5
  11. 2023Robust Visual Sim-to-Real Transfer for Robotic Manipulationcited by 5
  12. 2024Whole-body End-Effector Pose Trackingcited by 5
  13. 2023Dexterous In-hand Manipulation by Guiding Exploration with Simple Sub-skill Controllerscited by 3
  14. 2023Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real Worldcited by 3
  15. 2024RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Handscited by 3
  16. 2025Dynamic object goal pushing with mobile manipulators through model-free constrained reinforcement learningcited by 3
  17. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 3
  18. 2024Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learningcited by 2
  19. 2024Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledgecited by 2
  20. 2025Solving Robotics Tasks with Prior Demonstration via Exploration-Efficient Deep Reinforcement Learningcited by 1
  21. 2023Parallel Q-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulationcited by 0
  22. 2025No More Marching: Learning Humanoid Locomotion for Short-Range SE(2) Targetscited by 0