mlsanityData to 7 Oct 2026

Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

arXiv 2312.01853 ↗First posted 4 Dec 2023cs.RO cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. The method, trained in a simulated environment and then deployed to a real robot, is applicable to various in-hand object rotation tasks. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/ .

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 40 of its 68 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

40 of 68 references.

References · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  7. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  8. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  9. 2021A System for General In-Hand Object Re-Orientationcited by 155
  10. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  11. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  12. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
28 more
  1. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  2. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  3. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  4. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  5. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  6. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  7. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  8. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  9. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  10. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  11. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  12. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  13. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  14. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  15. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  16. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 33
  17. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
  18. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  19. 20183D Shape Perception from Monocular Vision, Touch, and Shape Priorscited by 31
  20. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 31
  21. 2022Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planningcited by 21
  22. 2022Surprisingly Robust In-Hand Manipulation: An Empirical Studycited by 16
  23. 2020In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulationcited by 9
  24. 2021On the Feasibility of Learning Finger-gaiting In-hand Manipulation with Intrinsic Sensingcited by 9
  25. 2022HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstrationcited by 8
  26. 2017Sampling-based Planning of In-Hand Manipulation with External Pushescited by 7
  27. 2023Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulationcited by 7
  28. 2022In-Hand Manipulation of Unknown Objects with Tactile Sensing for Insertioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 18; 2026: 16.

51 papers.

Sort:

By area:

  1. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  2. 2025Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulationcited by 103
  3. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  4. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 67
  5. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
  6. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  7. 2024Twisting Lids Off with Two Handscited by 36
  8. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  9. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  10. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  11. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  12. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
38 more
  1. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  2. 2024Lessons from Learning to Spin "Pens"cited by 22
  3. 2024AnySkin: Plug-and-play Skin Sensing for Robotic Touchcited by 22
  4. 2024AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touchcited by 18
  5. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 18
  6. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 13
  7. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 11
  8. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 10
  9. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 9
  10. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  11. 2025RUKA: Rethinking the Design of Humanoid Hands with Learningcited by 8
  12. 2025DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Modelcited by 8
  13. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  14. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  15. 2025eFlesh: Highly customizable Magnetic Touch Sensing using Cut-Cell Microstructurescited by 4
  16. 2026FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systemscited by 4
  17. 2025Composing Dextrous Grasping and In-hand Manipulation via Scoring with a Reinforcement Learning Criticcited by 3
  18. 2026Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learningcited by 3
  19. 2025VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulationcited by 2
  20. 2025Learning Adaptive Dexterous Grasping from Single Demonstrationscited by 2
  21. 2025Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motionscited by 2
  22. 2025RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AIcited by 2
  23. 2025DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulationcited by 2
  24. 2026Tactile Modality Fusion for Vision-Language-Action Modelscited by 2
  25. 2024Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understandingcited by 1
  26. 2025Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Settingcited by 1
  27. 2026roto 2.0: The Robot Tactile Olympiadcited by 1
  28. 2026Tac-DINO: Learning Vision-Tactile Features with Patch Alignmentcited by 1
  29. 2026Robust In-Hand Manipulation via Priors in Reinforcement Learning and Mechanical Designcited by 1
  30. 2026Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulationcited by 0
  31. 2026Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputscited by 0
  32. 2026TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objectscited by 0
  33. 2026Mana: Dexterous Manipulation of Articulated Toolscited by 0
  34. 2026NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulationcited by 0
  35. 2026CoStream: Composing Simple Behaviors for Generalizable Complex Manipulationcited by 0
  36. 2026VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulationcited by 0
  37. 2026WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulationcited by 0
  38. 2026AURORA: Active Uncertainty-Driven Re-Orientation for In-Hand Reconstructioncited by 0

50 most cited of 51 All 51 (JSON)