mlsanityData to 14 Oct 2026

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

arXiv 2410.24091 ↗First posted 31 Oct 2024cs.RO cs.AI cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Tactile and visual perception are both crucial for humans to perform fine-grained interactions with their environment. Developing similar multi-modal sensing capabilities for robots can significantly enhance and expand their manipulation skills. This paper introduces 3D-ViTac, a multi-modal sensing and learning system designed for dexterous bimanual manipulation. Our system features tactile sensors equipped with dense sensing units, each covering an area of 3mm2. These sensors are low-cost and flexible, providing detailed and extensive coverage of physical contacts, effectively complementing visual information. To integrate tactile and visual data, we fuse them into a unified 3D representation space that preserves their 3D structures and spatial relationships. The multi-modal representation can then be coupled with diffusion policies for imitation learning. Through concrete hardware experiments, we demonstrate that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies, particularly in safe interactions with fragile items and executing long-horizon tasks involving in-hand manipulation. Our project page is available at urlhttps://binghao-huang.github.io/3D-ViTac/.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

43 of 63 references.

References · 43

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  5. 2021Implicit Behavioral Cloningcited by 452
  6. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  7. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  8. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  9. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  10. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 167
  11. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
  12. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 154
31 more
  1. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 152
  2. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  3. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  4. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  5. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  6. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  7. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 78
  8. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 76
  9. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 67
  10. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 62
  11. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 56
  12. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 49
  13. 2022Visuo-Tactile Transformers for Manipulationcited by 47
  14. 2024Twisting Lids Off with Two Handscited by 42
  15. 2021GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Fingercited by 41
  16. 2023Stabilize to Act: Learning to Coordinate for Bimanual Manipulationcited by 39
  17. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 35
  18. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 31
  19. 2017Active Clothing Material Perception using Tactile Sensing and Deep Learningcited by 30
  20. 2020OmniTact: A Multi-Directional High Resolution Touch Sensorcited by 28
  21. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  22. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 28
  23. 2022Visuotactile Affordances for Cloth Manipulation with Local Controlcited by 21
  24. 2021Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translationcited by 19
  25. 2020Soft-Bubble grippers for robust and perceptive manipulationcited by 18
  26. 2021Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulationcited by 14
  27. 2022TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensorscited by 12
  28. 2023GelSight Svelte: A Human Finger-shaped Single-camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensingcited by 11
  29. 2024Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learningcited by 11
  30. 2021Improving Grasp Stability with Rotation Measurement from Tactile Sensingcited by 6
  31. 2021SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Usecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 26; 2026: 45.

72 papers.

Sort:

By area:

  1. 2025Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulationcited by 108
  2. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 76
  3. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  4. 2025VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedbackcited by 38
  5. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 38
  6. 2025TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Modelscited by 32
  7. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  8. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulationcited by 24
  9. 2025Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulationcited by 19
  10. 2026TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulationcited by 19
  11. 2026VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAscited by 17
  12. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
38 more
  1. 2026FAVLA: A Force-Adaptive Fast-Slow VLA model for Contact-Rich Robotic Manipulationcited by 15
  2. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  3. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 13
  4. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 11
  5. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  6. 2025OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfercited by 10
  7. 2025End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collectioncited by 9
  8. 2026OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfercited by 6
  9. 2026FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systemscited by 6
  10. 2025eFlesh: Highly customizable Magnetic Touch Sensing using Cut-Cell Microstructurescited by 5
  11. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 5
  12. 2026Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLAcited by 5
  13. 2026UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Modelscited by 5
  14. 2025On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lightingcited by 4
  15. 2025DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulationcited by 4
  16. 2026Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategycited by 4
  17. 2026ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulationcited by 4
  18. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environmentscited by 3
  19. 2026TacCoRL: Integrating Tactile Feedback into VLA via Simulationcited by 3
  20. 2026Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learningcited by 3
  21. 2026FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulationcited by 3
  22. 2025MILE: A Mechanically Isomorphic Hand Exoskeleton and Visuotactile Robotic Hand for Data Collection in Dexterous Manipulationcited by 2
  23. 2026Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulationcited by 2
  24. 2026Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlationcited by 2
  25. 2026SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)cited by 2
  26. 2026TacForcing: Streaming Action Generation with Execution-Time Tactile Feedbackcited by 2
  27. 2026TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulationcited by 2
  28. 2026Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Handscited by 2
  29. 2024ALPHA-α and Bi-ACT Are All You Need: Importance of Position and Force Information/Control for Imitation Learning of Unimanual and Bimanual Robotic Manipulation with Low-Cost Systemcited by 1
  30. 2026DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulationcited by 1
  31. 2026HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learningcited by 1
  32. 2026Tac-DINO: Learning Vision-Tactile Features with Patch Alignmentcited by 1
  33. 2026CoStream: Composing Simple Behaviors for Generalizable Complex Manipulationcited by 1
  34. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  35. 2026Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learningcited by 1
  36. 2026Robust Bimanual Vision-Language-Action Models via Embarrassingly Simple Modality Maskingcited by 1
  37. 2025ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generationcited by 0
  38. 2025Learning Dexterous Object Handovercited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
  2. 2025PolyTouch: A Robust Multi-Modal Tactile Sensor for Contact-rich Manipulation Using Tactile-Diffusion Policies
  3. 2026Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation ↗
  4. 2023Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations
  5. 2026ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
  6. 2026TacO: Benchmarking Tactile Sensors for Object Manipulation
  7. 2025Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
  8. 2026Tactile-Guided Robot Learning: From Representation Learning to Foundation Models ↗