mlsanityData to 7 Oct 2026

Visuo-Tactile Transformers for Manipulation

arXiv 2210.00121 ↗First posted 30 Sep 2022cs.RO cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer citedosovitskiy2021image to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the visual domain. We demonstrate the efficacy of VTT for representation learning with a comparative evaluation against baselines on four simulated robot tasks and one real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross-modality in representation learning.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

38 of 48 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018World Modelscited by 1,444
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  11. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  12. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
26 more
  1. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  2. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  3. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  4. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  5. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  6. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  7. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  8. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  9. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  10. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  11. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 80
  12. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  13. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  14. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  15. 2021Partially Observable Markov Decision Processes (POMDPs) and Roboticscited by 49
  16. 2021Coupling Vision and Proprioception for Navigation of Legged Robotscited by 46
  17. 2021GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Fingercited by 39
  18. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  19. 2021Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projectionscited by 27
  20. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27
  21. 2022VIRDO: Visio-tactile Implicit Representations of Deformable Objectscited by 25
  22. 2019Imitation Learning from Video by Leveraging Proprioceptioncited by 23
  23. 2019Multi-modality Latent Interaction Network for Visual Question Answeringcited by 23
  24. 2019Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulationcited by 15
  25. 2018Learning Awareness Modelscited by 13
  26. 2021Touch-based Curiosity for Sparse-Reward Taskscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 10; 2026: 21.

43 papers.

Sort:

By area:

  1. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  2. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 67
  3. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  4. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  5. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 30
  6. 2025TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Modelscited by 30
  7. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 29
  8. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  9. 2024AnySkin: Plug-and-play Skin Sensing for Robotic Touchcited by 22
  10. 2024VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policiescited by 14
  11. 2026TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulationcited by 11
  12. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 9
31 more
  1. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 7
  2. 2025ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representationscited by 7
  3. 2024Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assemblycited by 6
  4. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  5. 2026FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systemscited by 4
  6. 2024Precision-Focused Reinforcement Learning Model for Robotic Object Pushingcited by 3
  7. 2025Multimodal Diffusion Forcing for Forceful Manipulationcited by 3
  8. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 3
  9. 2026CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Expertscited by 3
  10. 2024AugInsert: Learning Robust Visual-Force Policies via Data Augmentation for Object Assembly Taskscited by 2
  11. 2025VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulationcited by 2
  12. 2025ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformerscited by 2
  13. 2025Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learningcited by 2
  14. 2026Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigmscited by 2
  15. 2026Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlationcited by 2
  16. 2026FlowTouch: View-Invariant Visuo-Tactile Predictioncited by 1
  17. 2026DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulationcited by 1
  18. 2025DML-RAM: Deep Multimodal Learning Framework for Robotic Arm Manipulation using Pre-trained Modelscited by 0
  19. 2026Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wristcited by 0
  20. 2026RAPID: Reconfigurable, Adaptive Platform for Iterative Designcited by 0
  21. 2026Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonancecited by 0
  22. 2026Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulationcited by 0
  23. 2026Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representationcited by 0
  24. 2026Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulationcited by 0
  25. 2026NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulationcited by 0
  26. 2026TAP-VLA: Tactile Annotation Prompting for Vision Language Action Modelscited by 0
  27. 2026TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulationcited by 0
  28. 2026S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Informationcited by 0
  29. 2026VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulationcited by 0
  30. 2026TAO-Force: Unifying Force-Aware Perception and Fast-Slow Control for Contact-Rich Manipulationcited by 0
  31. 2026What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memorycited by 0