mlsanityData to 7 Oct 2026

General In-Hand Object Rotation with Vision and Touch

arXiv 2309.09979 ↗First posted 18 Sep 2023cs.RO cs.AI cs.CVcited by 87

Not on the map. Why?

Abstract (arXiv)

We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 61 of its 86 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

61 of 86 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Segment Anythingcited by 8,158
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  10. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  11. 2019Solving Rubik's Cube with a Robot Handcited by 699
  12. 2018Learning Dexterous In-Hand Manipulationcited by 643
38 more
  1. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  2. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  3. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2019Learning by Cheatingcited by 361
  6. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  7. 2022Masked Visual Pre-training for Motor Controlcited by 246
  8. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  9. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  10. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  11. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  12. 2021A System for General In-Hand Object Re-Orientationcited by 155
  13. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  14. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  15. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  16. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  17. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  18. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  19. 2021Learning High-Speed Flight in the Wildcited by 98
  20. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 87
  21. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  22. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  23. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  24. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  25. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  26. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  27. 2020EGAD! an Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulationcited by 61
  28. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  29. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  30. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 49
  31. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  32. 20203D Shape Reconstruction from Vision and Touchcited by 42
  33. 2016Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstrationcited by 35
  34. 2021Active 3D Shape Reconstruction from Vision and Touchcited by 34
  35. 2022MidasTouch: Monte-Carlo inference over distributions across sliding touchcited by 33
  36. 2021ShapeMap 3-D: Efficient shape mapping through dense touch and visioncited by 30
  37. 2022Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planningcited by 21
  38. 2022Visuotactile Affordances for Cloth Manipulation with Local Controlcited by 20

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 27; 2025: 33; 2026: 23.

87 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  3. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  4. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  5. 2025Flow Matching Policy Gradientscited by 86
  6. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
  7. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  8. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  9. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  10. 2025TLA: Tactile-Language-Action Model for Contact-Rich Manipulationcited by 46
  11. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  12. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
38 more
  1. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  2. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40
  3. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  4. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  5. 2024Twisting Lids Off with Two Handscited by 36
  6. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
  7. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  8. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  9. 2024Tactile-Augmented Radiance Fieldscited by 26
  10. 2024Body Transformer: Leveraging Robot Embodiment for Policy Learningcited by 26
  11. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  12. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  13. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  14. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 24
  15. 2024Lessons from Learning to Spin "Pens"cited by 22
  16. 2024AnySkin: Plug-and-play Skin Sensing for Robotic Touchcited by 22
  17. 2024AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touchcited by 18
  18. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 18
  19. 2024Touch-GS: Visual-Tactile Supervised 3D Gaussian Splattingcited by 16
  20. 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Datacited by 15
  21. 2025DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulationcited by 15
  22. 2026Flow Policy Gradients for Robot Controlcited by 14
  23. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 13
  24. 2024Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfacescited by 12
  25. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 11
  26. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 10
  27. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 10
  28. 2026Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulationcited by 10
  29. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 9
  30. 2024Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulationcited by 8
  31. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  32. 2025DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Modelcited by 8
  33. 2024DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robotscited by 7
  34. 2024Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Datasetcited by 6
  35. 2025Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulationcited by 6
  36. 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learningcited by 5
  37. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 5
  38. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5

50 most cited of 87 All 87 (JSON)