# Demonstrating the Octopi-1.5 Visual-Tactile-Language Model

arXiv: https://arxiv.org/abs/2507.09985 · HTML: https://mlsanity.com/paper/2507.09985/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)

## Built on by (most cited first, 9 of 9)

- [OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2603.19201/) (2026)
- [TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation](https://mlsanity.com/paper/2603.12665/) (2026)
- [TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation](https://mlsanity.com/paper/2601.20321/) (2026)
- [SkillVLA: Tackling Combinatorial Diversity in Dual-Arm Manipulation via Skill Reuse](https://mlsanity.com/paper/2603.03836/) (2026)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation](https://mlsanity.com/paper/2606.11637/) (2026)
- [TacReasoner: A Dynamic Tactile-Language Framework for Interactive Reasoning in Real-World Scenarios](https://mlsanity.com/paper/2607.05131/) (2026)
- [GeoTLM: Geometry-aware Tactile-Language Models for Contact Motion Orientation Reasoning of Dynamic Objects](https://mlsanity.com/paper/2606.15909/) (2026)
- [Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework](https://mlsanity.com/paper/2609.13335/) (2026)

## Abstract (arXiv)

Touch is recognized as a vital sense for humans and an equally important modality for robots, especially for dexterous manipulation, material identification, and scenarios involving visual occlusion. Building upon very recent work in touch foundation models, this demonstration will feature Octopi-1.5, our latest visual-tactile-language model. Compared to its predecessor, Octopi-1.5 introduces the ability to process tactile signals from multiple object parts and employs a simple retrieval-augmented generation (RAG) module to improve performance on tasks and potentially learn new objects on-the-fly. The system can be experienced live through a new handheld tactile-enabled interface, the TMI, equipped with GelSight and TAC-02 tactile sensors. This convenient and accessible setup allows users to interact with Octopi-1.5 without requiring a robot. During the demonstration, we will showcase Octopi-1.5 solving tactile inference tasks by leveraging tactile inputs and commonsense knowledge. For example, in a Guessing Game, Octopi-1.5 will identify objects being grasped and respond to follow-up queries about how to handle it (e.g., recommending careful handling for soft fruits). We also plan to demonstrate Octopi-1.5's RAG capabilities by teaching it new items. With live interactions, this demonstration aims to highlight both the progress and limitations of VTLMs such as Octopi-1.5 and to foster further interest in this exciting field. Code for Octopi-1.5 and design files for the TMI gripper are available at https://github.com/clear-nus/octopi-1.5.