mlsanityData to 7 Oct 2026

Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset

arXiv 2403.09813 ↗First posted 14 Mar 2024cs.CV cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Tactility provides crucial support and enhancement for the perception and interaction capabilities of both humans and robots. Nevertheless, the multimodal research related to touch primarily focuses on visual and tactile modalities, with limited exploration in the domain of language. Beyond vocabulary, sentence-level descriptions contain richer semantics. Based on this, we construct a touch-language-vision dataset named TLV (Touch-Language-Vision) by human-machine cascade collaboration, featuring sentence-level descriptions for multimode alignment. The new dataset is used to fine-tune our proposed lightweight training framework, STLV-Align (Synergistic Touch-Language-Vision Alignment), achieving effective semantic alignment with minimal parameter adjustments (1%). Project Page: https://xiaoen0.github.io/touch.page/.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

33 of 44 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  10. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  11. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  12. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
21 more
  1. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  2. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  3. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  4. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  5. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  6. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  7. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  8. 2022Can Language Understand Depth?cited by 66
  9. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 65
  10. 2021Taxim: An Example-based Simulation Model for GelSight Tactile Sensorscited by 42
  11. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  12. 2022Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Featurescited by 37
  13. 2022MidasTouch: Monte-Carlo inference over distributions across sliding touchcited by 33
  14. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 30
  15. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  16. 2017Improved GelSight Tactile Sensor for Measuring Geometry and Slipcited by 27
  17. 2023Generating Visual Scenes from Touchcited by 26
  18. 2017Connecting Look and Feel: Associating the visual and tactile properties of physical materialscited by 22
  19. 2017Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensorcited by 19
  20. 2019Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matchingcited by 16
  21. 2021Generation of GelSight Tactile Images for Sim2Real Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  2. 2025HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signalscited by 3
  3. 2025RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Datacited by 2
  4. 2026Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigmscited by 2
  5. 2025HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioningcited by 0
  6. 2026RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalizationcited by 0