mlsanityData to 7 Oct 2026

Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes

arXiv 2211.11744 ↗First posted 21 Nov 2022cs.RO cs.AI cs.CVcited by 92

Not on the map. Why?

Abstract (arXiv)

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments that remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasistatic manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints which make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real-time, with the median reorientation time being close to seven seconds. The controller is trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only uses open-source components that cost less than five thousand dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56 percent of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23 degrees) 75 percent of the time. Videos are available at: https://taochenshh.github.io/projects/visual-dexterity.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

33 of 56 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  7. 2019Solving Rubik's Cube with a Robot Handcited by 699
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  11. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  12. 2021Implicit Behavioral Cloningcited by 442
21 more
  1. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  2. 2019Learning by Cheatingcited by 361
  3. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  4. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  5. 2021A System for General In-Hand Object Re-Orientationcited by 155
  6. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  7. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  8. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  9. 2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
  10. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 53
  11. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 46
  12. 2021Learning to Jump from Pixelscited by 39
  13. 2021Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFingercited by 34
  14. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  15. 2020Learning Dexterous Manipulation from Suboptimal Expertscited by 23
  16. 2022Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planningcited by 21
  17. 2022Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Handcited by 17
  18. 2022Surprisingly Robust In-Hand Manipulation: An Empirical Studycited by 16
  19. 2018Relaxed-Rigidity Constraints: Kinematic Trajectory Optimization and Collision Avoidance for In-Grasp Manipulationcited by 11
  20. 2021On the Feasibility of Learning Finger-gaiting In-hand Manipulation with Intrinsic Sensingcited by 9
  21. 2021TrajectoTree: Trajectory Optimization Meets Tree Search for Planning Multi-contact Dexterous Manipulationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 33; 2025: 28; 2026: 22.

92 papers.

Sort:

By area:

  1. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  2. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  3. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  4. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  5. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  6. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  7. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  8. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
  9. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 67
  10. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  11. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  12. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
38 more
  1. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  2. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  3. 2024Twisting Lids Off with Two Handscited by 36
  4. 2023DribbleBot: Dynamic Legged Manipulation in the Wildcited by 33
  5. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  6. 2025DexMachina: Functional Retargeting for Bimanual Dexterous Manipulationcited by 30
  7. 2023RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learningcited by 29
  8. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 27
  9. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  10. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  11. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  12. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  13. 2024Lessons from Learning to Spin "Pens"cited by 22
  14. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedbackcited by 21
  15. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  16. 2024Learning Force Control for Legged Manipulationcited by 19
  17. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
  18. 2025Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomycited by 19
  19. 2024AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touchcited by 18
  20. 2024Neural MP: A Generalist Neural Motion Plannercited by 18
  21. 2023DEFT: Dexterous Fine-Tuning for Real-World Hand Policiescited by 15
  22. 2025DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulationcited by 15
  23. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 13
  24. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 13
  25. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  26. 2024DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexteritycited by 11
  27. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 10
  28. 2024Robot Learning with Super-Linear Scalingcited by 10
  29. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  30. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 10
  31. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  32. 2024Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulationcited by 8
  33. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  34. 2025DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Modelcited by 8
  35. 2024When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?cited by 7
  36. 2025A roadmap for AI in roboticscited by 7
  37. 2025Neural Robot Dynamicscited by 7
  38. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7

50 most cited of 92 All 92 (JSON)