# Octopi: Object Property Reasoning with Large Tactile-Language Models arXiv: https://arxiv.org/abs/2405.02794 ยท HTML: https://mlsanity.com/paper/2405.02794/ ## Key facts - Cited by 48 papers in the map (data through 14 Oct 2026). - Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. - Builds on 4 of its 63 references that are in the map. - First posted on arXiv 5 May 2024. ## Placed in - LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/) ## Builds on (most cited first, 4 of 4) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023) - [NEWTON: Are Large Language Models Capable of Physical Reasoning?](https://mlsanity.com/paper/2310.07018/) (2023) ## Built on by (most cited first, 30 of 48) - [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024) - [AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation](https://mlsanity.com/paper/2410.00371/) (2024) - [AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors](https://mlsanity.com/paper/2502.12191/) (2025) - [Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding](https://mlsanity.com/paper/2501.04693/) (2025) - [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025) - [VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback](https://mlsanity.com/paper/2507.17294/) (2025) - [OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing](https://mlsanity.com/paper/2508.08706/) (2025) - [OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2603.19201/) (2026) - [UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning](https://mlsanity.com/paper/2509.18094/) (2025) - [PUGS: Zero-shot Physical Understanding with Gaussian Splatting](https://mlsanity.com/paper/2502.12231/) (2025) - [ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation](https://mlsanity.com/paper/2505.09698/) (2025) - [TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation](https://mlsanity.com/paper/2601.20321/) (2026) - [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026) - [TextToucher: Fine-Grained Text-to-Touch Generation](https://mlsanity.com/paper/2409.05427/) (2024) - [Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments](https://mlsanity.com/paper/2409.20445/) (2024) - [Universal Visuo-Tactile Video Understanding for Embodied Interaction](https://mlsanity.com/paper/2505.22566/) (2025) - [Generative AI Agents in Autonomous Machines: A Safety Perspective](https://mlsanity.com/paper/2410.15489/) (2024) - [Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant](https://mlsanity.com/paper/2409.20548/) (2024) - [EmbRACE-3K: Embodied Reasoning and Action in Complex Environments](https://mlsanity.com/paper/2507.10548/) (2025) - [UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation](https://mlsanity.com/paper/2606.31451/) (2026) - [X-Capture: An Open-Source Portable Device for Multi-Sensory Learning](https://mlsanity.com/paper/2504.02318/) (2025) - [Robot Operation of Home Appliances by Reading User Manuals](https://mlsanity.com/paper/2505.20424/) (2025) - [Tactile Modality Fusion for Vision-Language-Action Models](https://mlsanity.com/paper/2603.14604/) (2026) - [SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarios](https://mlsanity.com/paper/2505.04201/) (2025) - [Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience](https://mlsanity.com/paper/2505.05515/) (2025) - [Tactile MNIST: Benchmarking Active Tactile Perception](https://mlsanity.com/paper/2506.06361/) (2025) - [RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data](https://mlsanity.com/paper/2505.14270/) (2025) - [Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?](https://mlsanity.com/paper/2510.16924/) (2025) - [TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Data](https://mlsanity.com/paper/2602.09893/) (2026) - [VitaTouch: Property-Aware Vision-Tactile-Language Model for Robotic Quality Inspection in Manufacturing](https://mlsanity.com/paper/2604.03322/) (2026) ## Abstract (arXiv) Physical reasoning is important for effective robot manipulation. Recent work has investigated both vision and language modalities for physical reasoning; vision can reveal information about objects in the environment and language serves as an abstraction and communication medium for additional context. Although these works have demonstrated success on a variety of physical reasoning tasks, they are limited to physical properties that can be inferred from visual or language inputs. In this work, we investigate combining tactile perception with language, which enables embodied systems to obtain physical properties through interaction and apply commonsense reasoning. We contribute a new dataset PhysiCLeAR, which comprises both physical/property reasoning tasks and annotated tactile videos obtained using a GelSight tactile sensor. We then introduce Octopi, a system that leverages both tactile representation learning and large vision-language models to predict and reason about tactile inputs with minimal language fine-tuning. Our evaluations on PhysiCLeAR show that Octopi is able to effectively use intermediate physical property predictions to improve its performance on various tactile-related tasks. PhysiCLeAR and Octopi are available at https://github.com/clear-nus/octopi.