mlsanityData to 14 Oct 2026

Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

arXiv 2303.12076 ↗First posted 21 Mar 2023cs.RO cs.AI cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

Teaching dexterity to multi-fingered robots has been a longstanding challenge in robotics. Most prominent work in this area focuses on learning controllers or policies that either operate on visual observations or state estimates derived from vision. However, such methods perform poorly on fine-grained manipulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-Dex, a new approach for tactile-based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self-supervised tactile encoders. This is necessary to bring high-dimensional tactile readings to a lower-dimensional embedding. In the second phase, given a handful of demonstrations for a dexterous task, we learn non-parametric policies that combine the tactile observations with visual ones. Across five challenging dexterous tasks, we show that our tactile-based dexterity models outperform purely vision and torque-based models by an average of 1.7X. Finally, we provide a detailed analysis on factors critical to T-Dex including the importance of play data, architectures, and representation learning.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 59 of its 90 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

59 of 90 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  5. 2018World Modelscited by 1,513
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2019Solving Rubik's Cube with a Robot Handcited by 734
  9. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  12. 2018Learning Dexterous In-Hand Manipulationcited by 687
38 more
  1. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  2. 2021Implicit Behavioral Cloningcited by 452
  3. 2019Learning Latent Plans from Playcited by 316
  4. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  5. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  7. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  8. 2021Reinforcement Learning with Prototypical Representationscited by 179
  9. 2021A System for General In-Hand Object Re-Orientationcited by 165
  10. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  11. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  12. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  13. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  14. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  15. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 136
  16. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 114
  17. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  18. 2020Visual Imitation Made Easycited by 111
  19. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 107
  20. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  21. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  22. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  23. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 84
  24. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 84
  25. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  26. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 78
  27. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 57
  28. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 50
  29. 2019Cable Manipulation with a Tactile-Reactive Grippercited by 46
  30. 2016The Curious Robot: Learning Visual Representations via Physical Interactionscited by 38
  31. 2022BYOL for Audio: Exploring Pre-trained General-purpose Audio Representationscited by 36
  32. 2021ReSkin: versatile, replaceable, lasting tactile skinscited by 35
  33. 2021GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Fingercited by 34
  34. 2020Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigationcited by 32
  35. 20183D Shape Perception from Monocular Vision, Touch, and Shape Priorscited by 31
  36. 2017Improved GelSight Tactile Sensor for Measuring Geometry and Slipcited by 28
  37. 2022Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Handcited by 20
  38. 2018Learning to Grasp Without Seeingcited by 19

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 18; 2025: 17; 2026: 15.

62 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  2. 2023A Cookbook of Self-Supervised Learningcited by 250
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  4. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 160
  5. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  6. 2023On Bringing Robots Homecited by 101
  7. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  8. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  9. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  10. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 76
  11. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 72
  12. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 56
38 more
  1. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  2. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  3. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  4. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 35
  5. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  6. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 34
  7. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 31
  8. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 31
  9. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  10. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  11. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 28
  12. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 26
  13. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 20
  14. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  15. 2025CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-Worldcited by 18
  16. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 18
  17. 2024Hierarchical State Space Models for Continuous Sequence-to-Sequence Modelingcited by 15
  18. 2024VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policiescited by 15
  19. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 14
  20. 2026T-Rex: Tactile-Reactive Dexterous Manipulationcited by 13
  21. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 12
  22. 2023ArrayBot: Reinforcement Learning for Generalizable Distributed Manipulation through Touchcited by 11
  23. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 8
  24. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  25. 2025Sensor-Invariant Tactile Representationcited by 8
  26. 2025RUKA: Rethinking the Design of Humanoid Hands with Learningcited by 8
  27. 2024HyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signals Through Contrastive Learningcited by 6
  28. 2026FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systemscited by 6
  29. 2023Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representationcited by 5
  30. 2025V-HOP: Visuo-Haptic 6D Object Pose Trackingcited by 5
  31. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  32. 2025ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimationcited by 3
  33. 2025Adaptive Wiping: Adaptive contact-rich manipulation through few-shot imitation learning with Force-Torque feedback and pre-trained object representationscited by 3
  34. 2025Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learningcited by 3
  35. 2026FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulationcited by 3
  36. 2025Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Settingcited by 1
  37. 2025Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learningcited by 1
  38. 2025Toward Artificial Palpation: Representation Learning of Touch on Soft Bodiescited by 1

50 most cited of 62 All 62 (JSON)