mlsanityData to 7 Oct 2026

A Touch, Vision, and Language Dataset for Multimodal Alignment

arXiv 2402.13232 ↗First posted 20 Feb 2024cs.CV cs.ROcited by 68

Not on the map. Why?

Abstract (arXiv)

Touch is an important sensing modality for humans, but it has not yet been incorporated into a multimodal generative language model. This is partially due to the difficulty of obtaining natural language labels for tactile data and the complexity of aligning tactile readings with both visual observations and language descriptions. As a step towards bridging that gap, this work introduces a new dataset of 44K in-the-wild vision-touch pairs, with English language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V (90%). We use this dataset to train a vision-language-aligned tactile encoder for open-vocabulary classification and a touch-vision-language (TVL) model for text generation using the trained encoder. Results suggest that by incorporating touch, the TVL model improves (+29% classification accuracy) touch-vision-language alignment over existing models trained on any pair of those modalities. Although only a small fraction of the dataset is human-labeled, the TVL model demonstrates improved visual-tactile understanding over GPT-4V (+12%) and open-source vision-language models (+32%) on a new touch-vision understanding benchmark. Code and data: https://tactile-vlm.github.io.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 44 of its 103 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

44 of 103 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
32 more
  1. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  11. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  12. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  13. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  14. 2020Long-tailed Recognition by Routing Diverse Distribution-Aware Expertscited by 270
  15. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  16. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  17. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  18. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  19. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  20. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  21. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  22. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  23. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  24. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  25. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  26. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  27. 2022Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Featurescited by 37
  28. 2022MidasTouch: Monte-Carlo inference over distributions across sliding touchcited by 33
  29. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 30
  30. 2017Active Clothing Material Perception using Tactile Sensing and Deep Learningcited by 29
  31. 2023Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Modelscited by 17
  32. 2022Safe Self-Supervised Learning in Real of Visuo-Tactile Feedback Policies for Industrial Insertioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 32; 2026: 25.

68 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  5. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  6. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  7. 2025TLA: Tactile-Language-Action Model for Contact-Rich Manipulationcited by 46
  8. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  9. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  10. 2025VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedbackcited by 37
  11. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 37
  12. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
38 more
  1. 2025Continual Multimodal Contrastive Learningcited by 19
  2. 2025Principled Multimodal Representation Learningcited by 19
  3. 2024Touch-GS: Visual-Tactile Supervised 3D Gaussian Splattingcited by 16
  4. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 15
  5. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 14
  6. 2025Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulationcited by 14
  7. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12
  8. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  9. 2024TextToucher: Fine-Grained Text-to-Touch Generationcited by 11
  10. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 11
  11. 2024Next Best Sense: Guiding Vision and Touch with FisherRF for 3D Gaussian Splattingcited by 10
  12. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  13. 2026T-Rex: Tactile-Reactive Dexterous Manipulationcited by 10
  14. 2025Demonstrating the Octopi-1.5 Visual-Tactile-Language Modelcited by 9
  15. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 7
  16. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 3
  17. 2025HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signalscited by 3
  18. 2025Calibrated Multimodal Representation Learning with Missing Modalitiescited by 3
  19. 2026DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adaptercited by 3
  20. 2026UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generationcited by 3
  21. 2024TouchAI: Exploring human-AI perceptual alignment in touch through language model representationscited by 2
  22. 2024Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Modelcited by 2
  23. 2025RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Datacited by 2
  24. 2025ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformerscited by 2
  25. 2025Tactile MNIST: Benchmarking Active Tactile Perceptioncited by 2
  26. 2025Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalitiescited by 2
  27. 2026Seeing Through Touch: Tactile-Driven Visual Localization of Material Regionscited by 2
  28. 2026Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigmscited by 2
  29. 2024LEMMo-Plan: LLM-Enhanced Learning from Multi-Modal Demonstration for Planning Sequential Contact-Rich Manipulation Taskscited by 1
  30. 2025Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Settingcited by 1
  31. 2025RETRO: REthinking Tactile Representation Learning with Material PriOrscited by 1
  32. 2025Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?cited by 1
  33. 2026FlowTouch: View-Invariant Visuo-Tactile Predictioncited by 1
  34. 2026VitaTouch: Property-Aware Vision-Tactile-Language Model for Robotic Quality Inspection in Manufacturingcited by 1
  35. 2026HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learningcited by 1
  36. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  37. 2025Advances in Compliance Detection: Novel Models Using Vision-Based Tactile Sensorscited by 0
  38. 2025MLLM-Fabric: Multimodal Large Language Model-Driven Robotic Framework for Fabric Sorting and Selectioncited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding
  2. 2026SimpleTouch: Can Vision-Language-Action Models Master Contact-Rich Manipulation Without Tactile Policy Pretraining?
  3. 2026TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation
  4. 2026τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision
  5. 2026FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation
  6. 2026Tactile Modality Fusion for Vision-Language-Action Models
  7. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarios
  8. 2026TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation