mlsanityData to 14 Oct 2026

Learning Visuotactile Skills with Two Multifingered Hands

arXiv 2404.16823 ↗First posted 25 Apr 2024cs.RO cs.AI cs.CVcited by 76

Not on the map. Why?

Abstract (arXiv)

Aiming to replicate human-like dexterity, perceptual experiences, and motion patterns, we explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing. To tackle the first challenge, we develop HATO, a low-cost hands-arms teleoperation system that leverages off-the-shelf electronics, complemented with a software suite that enables efficient data collection; the comprehensive software suite also supports multimodal data processing, scalable policy learning, and smooth policy deployment. To tackle the latter challenge, we introduce a novel hardware adaptation by repurposing two prosthetic hands equipped with touch sensors for research. Using visuotactile data collected from our system, we learn skills to complete long-horizon, high-precision tasks which are difficult to achieve without multifingered dexterity and touch feedback. Furthermore, we empirically investigate the effects of dataset size, sensing modality, and visual input preprocessing on policy learning. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data. Videos, code, and datasets can be found at https://toruowo.github.io/hato/ .

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

36 of 54 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  7. 2018Learning Dexterous In-Hand Manipulationcited by 687
  8. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  9. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  10. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  11. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulatorscited by 189
  12. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
24 more
  1. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  2. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  3. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 118
  4. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  5. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  6. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  7. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 82
  8. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 82
  9. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 78
  10. 2024OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulationcited by 71
  11. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 62
  12. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 57
  13. 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperationcited by 54
  14. 2023AirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wildcited by 47
  15. 2024Twisting Lids Off with Two Handscited by 42
  16. 2023Stabilize to Act: Learning to Coordinate for Bimanual Manipulationcited by 39
  17. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 35
  18. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 35
  19. 2021Active 3D Shape Reconstruction from Vision and Touchcited by 34
  20. 2021ShapeMap 3-D: Efficient shape mapping through dense touch and visioncited by 31
  21. 2023RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learningcited by 30
  22. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  23. 2022Visuotactile Affordances for Cloth Manipulation with Local Controlcited by 21
  24. 2022TANDEM3D: Active Tactile Exploration for 3D Object Recognitioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 36; 2026: 22.

76 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  2. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  3. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
  4. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 185
  5. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  6. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 145
  7. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  8. 2025Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulationcited by 108
  9. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  10. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 77
  11. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 76
  12. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 72
38 more
  1. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  2. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 52
  3. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 48
  4. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 44
  5. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  6. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 38
  7. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  8. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 31
  9. 2024Lessons from Learning to Spin "Pens"cited by 26
  10. 2024Bimanual Dexterity for Complex Taskscited by 25
  11. 2025Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loopcited by 25
  12. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  13. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  14. 2025Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalizationcited by 19
  15. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 15
  16. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  17. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 14
  18. 2026Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulationcited by 14
  19. 2026T-Rex: Tactile-Reactive Dexterous Manipulationcited by 13
  20. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 12
  21. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  22. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 11
  23. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 10
  24. 2025SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learningcited by 9
  25. 2025End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collectioncited by 9
  26. 2025Learning Dexterous Manipulation Skills from Imperfect Simulationscited by 9
  27. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  28. 2024Built Different: Tactile Perception to Overcome Cross-Embodiment Capability Differences in Collaborative Manipulationcited by 6
  29. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 6
  30. 2026FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systemscited by 6
  31. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 6
  32. 2025V-HOP: Visuo-Haptic 6D Object Pose Trackingcited by 5
  33. 2025D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentationcited by 5
  34. 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learningcited by 5
  35. 2025End-to-end RL Improves Dexterous Grasping Policiescited by 5
  36. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 5
  37. 2024RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Handscited by 4
  38. 2025DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulationcited by 4

50 most cited of 76 All 76 (JSON)