mlsanityData to 14 Oct 2026

Binding Touch to Everything: Learning Unified Multimodal Tactile Representations

arXiv 2401.18084 ↗First posted 31 Jan 2024cs.CV cs.ROcited by 95

Not on the map. Why?

Abstract (arXiv)

The ability to associate touch with other modalities has huge implications for humans and computational systems. However, multimodal learning with touch remains challenging due to the expensive data collection process and non-standardized sensor outputs. We introduce UniTouch, a unified tactile model for vision-based touch sensors connected to multiple modalities, including vision, language, and sound. We achieve this by aligning our UniTouch embeddings to pretrained image embeddings already associated with a variety of other modalities. We further propose learnable sensor-specific tokens, allowing the model to learn from a set of heterogeneous tactile sensors, all at the same time. UniTouch is capable of conducting various touch sensing tasks in the zero-shot setting, from robot grasping prediction to touch image question answering. To the best of our knowledge, UniTouch is the first to demonstrate such capabilities. Project page: https://cfeng16.github.io/UniTouch/

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Builds on 103 of its 147 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

103 of 147 references.

References · 103

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
38 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  2. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  3. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  7. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  8. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  9. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  10. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  11. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  12. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  13. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  14. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 167
  15. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  16. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  17. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  18. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  19. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  20. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 87
  21. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 85
  22. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 84
  23. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  24. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  25. 2022Can Language Understand Depth?cited by 69
  26. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 68
  27. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 68
  28. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 67
  29. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  30. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  31. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 63
  32. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 62
  33. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  34. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  35. 2021Taxim: An Example-based Simulation Model for GelSight Tactile Sensorscited by 43
  36. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Textscited by 42
  37. 2021GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Fingercited by 41
  38. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 39; 2026: 35.

95 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 69
  3. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 48
  4. 2025TLA: Tactile-Language-Action Model for Contact-Rich Manipulationcited by 47
  5. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  6. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 38
  7. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
  8. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  9. 2024Test-Time Adaptation for Depth Completioncited by 26
  10. 2024Tactile-Augmented Radiance Fieldscited by 26
  11. 2024RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptionscited by 21
  12. 2025RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learningcited by 20
38 more
  1. 2025Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulationcited by 18
  2. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  3. 2026VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAscited by 17
  4. 2024Touch-GS: Visual-Tactile Supervised 3D Gaussian Splattingcited by 16
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  6. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 15
  7. 2024Efficient Vision-Language Pre-training by Cluster Maskingcited by 13
  8. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 12
  9. 2024TextToucher: Fine-Grained Text-to-Touch Generationcited by 11
  10. 2025ProtoDepth: Unsupervised Continual Depth Completion with Prototypescited by 11
  11. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understandingcited by 11
  12. 2024Next Best Sense: Guiding Vision and Touch with FisherRF for 3D Gaussian Splattingcited by 10
  13. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  14. 2025Demonstrating the Octopi-1.5 Visual-Tactile-Language Modelcited by 9
  15. 2025Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokenscited by 9
  16. 2025Sensor-Invariant Tactile Representationcited by 8
  17. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 8
  18. 2025ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representationscited by 8
  19. 2024HyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signals Through Contrastive Learningcited by 6
  20. 2024Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generationcited by 6
  21. 2025Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbolscited by 6
  22. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 6
  23. 2024Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generationcited by 5
  24. 2024Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representationscited by 5
  25. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  26. 2025GPS as a Control Signal for Image Generationcited by 5
  27. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 5
  28. 2026UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generationcited by 5
  29. 2025Tacchi 2.0: A Low Computational Cost and Comprehensive Dynamic Contact Simulator for Vision-based Tactile Sensorscited by 4
  30. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 4
  31. 2025Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstructioncited by 4
  32. 2025ETA: Energy-based Test-time Adaptation for Depth Completioncited by 4
  33. 2025Seeing without Pixels: Perception from Camera Trajectoriescited by 4
  34. 2026DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adaptercited by 4
  35. 2026Tactile Modality Fusion for Vision-Language-Action Modelscited by 4
  36. 2025ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimationcited by 3
  37. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  38. 2026TacCoRL: Integrating Tactile Feedback into VLA via Simulationcited by 3

50 most cited of 95 All 95 (JSON)