mlsanityData to 7 Oct 2026

Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model

arXiv 2409.09276 ↗First posted 14 Sep 2024cs.ROcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Tactile perception is vital, especially when distinguishing visually similar objects. We propose an approach to incorporate tactile data into a Vision-Language Model (VLM) for visuo-tactile zero-shot object recognition. Our approach leverages the zero-shot capability of VLMs to infer tactile properties from the names of tactilely similar objects. The proposed method translates tactile data into a textual description solely by annotating object names for each tactile sequence during training, making it adaptable to various contexts with low training costs. The proposed method was evaluated on the FoodReplica and Cube datasets, demonstrating its effectiveness in recognizing objects that are difficult to distinguish by vision alone.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 36 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

17 of 36 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  8. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  9. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  10. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  11. 2017Active Clothing Material Perception using Tactile Sensing and Deep Learningcited by 29
  12. 2022Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generationcited by 28
5 more
  1. 2017Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensorcited by 19
  2. 2015Deep Learning for Tactile Understanding From Visual and Haptic Datacited by 18
  3. 2018Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Imagescited by 9
  4. 2021Playing with Food: Learning Food Item Representations through Interactive Explorationcited by 8
  5. 2021Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Lossescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  2. 2026When Sensing Varies with Contexts: Context Probing for Tactile Few-Shot Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing
  2. 2025Universal Visuo-Tactile Video Understanding for Embodied Interaction
  3. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
  4. 2025HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction ↗
  5. 2026TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models
  6. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  7. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
  8. 2026N0-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens