mlsanityData to 14 Oct 2026

See to Touch: Learning Tactile Dexterity through Visual Incentives

arXiv 2309.12300 ↗First posted 21 Sep 2023cs.RO cs.AI cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Equipping multi-fingered robots with tactile sensing is crucial for achieving the precise, contact-rich, and dexterous manipulation that humans excel at. However, relying solely on tactile sensing fails to provide adequate cues for reasoning about objects' spatial configurations, limiting the ability to correct errors and adapt to changing situations. In this paper, we present Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances tactile-based dexterity by optimizing dexterous policies using vision-based rewards. First, we use a contrastive-based objective to learn visual representations. Next, we construct a reward function using these visual representations through optimal-transport based matching on one human demonstration. Finally, we use online reinforcement learning on our robot to optimize tactile-based policies that maximize the visual reward. On six challenging tasks, such as peg pick-and-place, unstacking bowls, and flipping slender objects, TAVI achieves a success rate of 73% using our four-fingered Allegro robot hand. The increase in performance is 108% higher than policies using tactile and vision-based rewards and 135% higher than policies without tactile observational input. Robot videos are best viewed on our project website: https://see-to-touch.github.io/.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 52 of its 73 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

52 of 73 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2018Learning Dexterous In-Hand Manipulationcited by 687
38 more
  1. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  2. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  5. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  6. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  7. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  9. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  10. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  11. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  12. 2020Visual Imitation Made Easycited by 111
  13. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  14. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  15. 2019Recent Advances in Imitation Learning from Observationcited by 93
  16. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  17. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 78
  18. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 75
  19. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
  20. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 67
  21. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 62
  22. 2019Wasserstein Adversarial Imitation Learningcited by 60
  23. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 57
  24. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  25. 2019Cable Manipulation with a Tactile-Reactive Grippercited by 46
  26. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  27. 2017A survey of robot learning from demonstrations for Human-Robot Collaborationcited by 38
  28. 2021ReSkin: versatile, replaceable, lasting tactile skinscited by 35
  29. 2021GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Fingercited by 34
  30. 2020The MAGICAL Benchmark for Robust Imitationcited by 32
  31. 20183D Shape Perception from Monocular Vision, Touch, and Shape Priorscited by 31
  32. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 31
  33. 2017Improved GelSight Tactile Sensor for Measuring Geometry and Slipcited by 28
  34. 2021Domain-Robust Visual Imitation Learning with Mutual Information Constraintscited by 20
  35. 2018Learning to Grasp Without Seeingcited by 19
  36. 2019Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulationcited by 16
  37. 2021Learning rich touch representations through cross-modal self-supervisioncited by 13
  38. 2013A Compliant, Underactuated Hand for Robust Manipulationcited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 17; 2026: 7.

35 papers.

Sort:

By area:

  1. 2025Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulationcited by 108
  2. 2023On Bringing Robots Homecited by 101
  3. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  4. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 76
  5. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 56
  6. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  7. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
  8. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  9. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 29
  10. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 28
  11. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 26
  12. 2024Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalitiescited by 24
23 more
  1. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  2. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  3. 2025CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-Worldcited by 18
  4. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  5. 2024Hierarchical State Space Models for Continuous Sequence-to-Sequence Modelingcited by 15
  6. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  7. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 14
  8. 2024Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skinscited by 12
  9. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 11
  10. 2025RUKA: Rethinking the Design of Humanoid Hands with Learningcited by 8
  11. 2024Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignmentcited by 7
  12. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  13. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  14. 2025VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulationcited by 3
  15. 2026TacForcing: Streaming Action Generation with Execution-Time Tactile Feedbackcited by 2
  16. 2026CoStream: Composing Simple Behaviors for Generalizable Complex Manipulationcited by 1
  17. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 1
  18. 2025Tac2Motion: Contact-Aware Reinforcement Learning with Tactile Feedback for Robotic Hand Manipulationcited by 0
  19. 2025D3Grasp: Diverse and Deformable Dexterous Grasping for General Objectscited by 0
  20. 2026TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objectscited by 0
  21. 2026Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assemblycited by 0
  22. 2026DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imaginationcited by 0
  23. 2026Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulationcited by 0