mlsanityData to 14 Oct 2026

Octopi: Object Property Reasoning with Large Tactile-Language Models

arXiv 2405.02794 ↗First posted 5 May 2024cs.ROcited by 48

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Physical reasoning is important for effective robot manipulation. Recent work has investigated both vision and language modalities for physical reasoning; vision can reveal information about objects in the environment and language serves as an abstraction and communication medium for additional context. Although these works have demonstrated success on a variety of physical reasoning tasks, they are limited to physical properties that can be inferred from visual or language inputs. In this work, we investigate combining tactile perception with language, which enables embodied systems to obtain physical properties through interaction and apply commonsense reasoning. We contribute a new dataset PhysiCLeAR, which comprises both physical/property reasoning tasks and annotated tactile videos obtained using a GelSight tactile sensor. We then introduce Octopi, a system that leverages both tactile representation learning and large vision-language models to predict and reason about tactile inputs with minimal language fine-tuning. Our evaluations on PhysiCLeAR show that Octopi is able to effectively use intermediate physical property predictions to improve its performance on various tactile-related tasks. PhysiCLeAR and Octopi are available at https://github.com/clear-nus/octopi.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 4 of its 63 references that are in the map. First posted on arXiv 5 May 2024.

Builds on

4 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?10 Oct

Other references · 3

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 20; 2026: 22.

48 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 125
  3. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 48
  4. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  5. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
  6. 2025VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedbackcited by 38
  7. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 38
  8. 2026OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulationcited by 36
  9. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 28
  10. 2025PUGS: Zero-shot Physical Understanding with Gaussian Splattingcited by 24
  11. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  12. 2026TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulationcited by 13
36 more
  1. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 12
  2. 2024TextToucher: Fine-Grained Text-to-Touch Generationcited by 11
  3. 2024Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environmentscited by 9
  4. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 8
  5. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  6. 2024Robi Butler: Multimodal Remote Interaction with a Household Robot Assistantcited by 6
  7. 2025EmbRACE-3K: Embodied Reasoning and Action in Complex Environmentscited by 5
  8. 2026UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generationcited by 5
  9. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 4
  10. 2025Robot Operation of Home Appliances by Reading User Manualscited by 4
  11. 2026Tactile Modality Fusion for Vision-Language-Action Modelscited by 4
  12. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  13. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  14. 2025Tactile MNIST: Benchmarking Active Tactile Perceptioncited by 3
  15. 2025RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Datacited by 2
  16. 2025Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?cited by 2
  17. 2026TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Datacited by 2
  18. 2026VitaTouch: Property-Aware Vision-Tactile-Language Model for Robotic Quality Inspection in Manufacturingcited by 2
  19. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  20. 2026Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigmscited by 2
  21. 2026TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representationcited by 2
  22. 2026TacReasoner: A Dynamic Tactile-Language Framework for Interactive Reasoning in Real-World Scenarioscited by 2
  23. 2026SiPhy: Single-Image Physical Property Reasoningcited by 2
  24. 2025Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labelingcited by 1
  25. 2025Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inferencecited by 1
  26. 2026Touch-R1: Reinforcing Touch Reasoning in MLLMscited by 1
  27. 2026Adaptive Volumetric Mechanical Property Fields Invariant to Resolutioncited by 1
  28. 2025MLLM-Fabric: Multimodal Large Language Model-Driven Robotic Framework for Fabric Sorting and Selectioncited by 0
  29. 2026PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Modelscited by 0
  30. 2026StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoningcited by 0
  31. 2026GeoTLM: Geometry-aware Tactile-Language Models for Contact Motion Orientation Reasoning of Dynamic Objectscited by 0
  32. 2026RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalizationcited by 0
  33. 2026What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agentscited by 0
  34. 2026Language-Guided Representation Learning for Robust Cross-Sensor Material Recognitioncited by 0
  35. 2026ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perceptioncited by 0
  36. 2026PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization
  2. 2025Demonstrating the Octopi-1.5 Visual-Tactile-Language Model
  3. 2023Multi-Modal World Model for Physical Robot Interactions: Simultaneous Visual and Tactile Predictions for Enhanced Accuracy
  4. 2026Tactile-Guided Robot Learning: From Representation Learning to Foundation Models ↗
  5. 2026UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models
  6. 2026TacO: Benchmarking Tactile Sensors for Object Manipulation
  7. 2025Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation
  8. 2026ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations ↗