mlsanityData to 14 Oct 2026

AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch

arXiv 2405.07391 ↗First posted 12 May 2024cs.RO cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a dense tactile policy in simulation and present a sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects of varying properties. Interestingly, we found rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy. The project website can be found at https://maxyang27896.github.io/anyrotate/.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 12 May 2024.

Builds on

41 of 67 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  4. 2019Solving Rubik's Cube with a Robot Handcited by 734
  5. 2018Learning Dexterous In-Hand Manipulationcited by 687
  6. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  7. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  8. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  9. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 167
  10. 2021A System for General In-Hand Object Re-Orientationcited by 165
  11. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  12. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 147
29 more
  1. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 118
  2. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  3. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  4. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  5. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 63
  6. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 56
  7. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 49
  8. 2021Taxim: An Example-based Simulation Model for GelSight Tactile Sensorscited by 43
  9. 2021Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFingercited by 38
  10. 2021Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projectionscited by 28
  11. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  12. 2022Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planningcited by 26
  13. 2023Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulationcited by 24
  14. 2022Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Handcited by 20
  15. 2023Dextrous Tactile In-Hand Manipulation Using a Modular Reinforcement Learning Architecturecited by 20
  16. 2021Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translationcited by 19
  17. 2022Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touchcited by 18
  18. 2023Bi-Touch: Bimanual Tactile Manipulation with Sim-to-Real Deep Reinforcement Learningcited by 17
  19. 2021Generation of GelSight Tactile Images for Sim2Real Learningcited by 15
  20. 2021Soft Biomimetic Optical Tactile Sensing with the TacTip: A Reviewcited by 14
  21. 2021On the Feasibility of Learning Finger-gaiting In-hand Manipulation with Intrinsic Sensingcited by 11
  22. 2022DigiTac: A DIGIT-TacTip Hybrid Tactile Sensor for Comparing Low-Cost High-Resolution Robot Touchcited by 10
  23. 2023Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushingcited by 10
  24. 2023Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulationcited by 9
  25. 2023Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensorscited by 8
  26. 2018Geometric In-Hand Regrasp Planning: Alternating Optimization of Finger Gaits and In-Grasp Manipulationcited by 7
  27. 2021Reducing Tactile Sim2Real Domain Gaps via Deep Texture Generation Networkscited by 6
  28. 2023VERN: Vegetation-aware Robot Navigation in Dense Unstructured Outdoor Environmentscited by 6
  29. 2020Grasping and Manipulation with a Multi-Fingered Handcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 10.

21 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  3. 2024Lessons from Learning to Spin "Pens"cited by 26
  4. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 15
  5. 2025DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Modelcited by 9
  6. 2025Learning Dexterous Manipulation Skills from Imperfect Simulationscited by 9
  7. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  8. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  9. 2026Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learningcited by 3
  10. 2025Learning Long-Horizon Robot Manipulation Skills via Privileged Actioncited by 2
  11. 2025Apple: Toward General Active Perception via Reinforcement Learningcited by 2
  12. 2026ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientationcited by 2
9 more
  1. 2025Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functionscited by 1
  2. 2026roto 2.0: The Robot Tactile Olympiadcited by 1
  3. 2026Robust In-Hand Manipulation via Priors in Reinforcement Learning and Mechanical Designcited by 1
  4. 2026DexMani: Human-Derived Manipulability Guidance for Dexterous Rotationcited by 1
  5. 2026HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policiescited by 0
  6. 2026Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representationcited by 0
  7. 2026Mana: Dexterous Manipulation of Articulated Toolscited by 0
  8. 2026NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulationcited by 0
  9. 2026WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulationcited by 0