# Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation

arXiv: https://arxiv.org/abs/2512.09851 · HTML: https://mlsanity.com/paper/2512.09851/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [General In-Hand Object Rotation with Vision and Touch](https://mlsanity.com/paper/2309.09979/) (2023)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense](https://mlsanity.com/paper/2004.09044/) (2020)
- [FreeZe: Training-free zero-shot 6D pose estimation with geometric and vision foundation models](https://mlsanity.com/paper/2312.00947/) (2023)
- [Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks](https://mlsanity.com/paper/2406.13640/) (2024)
- [ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation](https://mlsanity.com/paper/2506.15953/) (2025)
- [Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation](https://mlsanity.com/paper/2503.10845/) (2025)
- [ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface](https://mlsanity.com/paper/2504.06156/) (2025)
- [9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation](https://mlsanity.com/paper/2308.14277/) (2023)
- [Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing](https://mlsanity.com/paper/2407.07885/) (2024)
- [UniT: Data Efficient Tactile Representation with Generalization to Unseen Objects](https://mlsanity.com/paper/2408.06481/) (2024)
- [PolyTouch: A Robust Multi-Modal Tactile Sensor for Contact-rich Manipulation Using Tactile-Diffusion Policies](https://mlsanity.com/paper/2504.19341/) (2025)
- [Touch begins where vision ends: Generalizable policies for contact-rich manipulation](https://mlsanity.com/paper/2506.13762/) (2025)
- [Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor](https://mlsanity.com/paper/2311.01248/) (2023)
- [Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects](https://mlsanity.com/paper/2403.01694/) (2024)
- [Embedding high-resolution touch across robotic hands enables adaptive human-like grasping](https://mlsanity.com/paper/2412.14482/) (2024)
- [Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation](https://mlsanity.com/paper/2504.12908/) (2025)
- [ViTacTip: Design and Verification of a Novel Biomimetic Physical Vision-Tactile Fusion Sensor](https://mlsanity.com/paper/2402.00199/) (2024)

## Built on by (most cited first, 6 of 6)

- [TacCoRL: Integrating Tactile Feedback into VLA via Simulation](https://mlsanity.com/paper/2606.11743/) (2026)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation](https://mlsanity.com/paper/2606.29941/) (2026)
- [Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance](https://mlsanity.com/paper/2605.14571/) (2026)
- [S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information](https://mlsanity.com/paper/2607.26047/) (2026)
- [ForeTac-VLA: A Forecasting-Based Tactile-Vision-Language-Action Model for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2609.20980/) (2026)

## Abstract (arXiv)

Robotic manipulation requires both rich multimodal perception and effective learning frameworks to handle complex real-world tasks. See-through-skin (STS) sensors, which combine tactile and visual perception, offer promising sensing capabilities, while modern imitation learning provides powerful tools for policy acquisition. However, existing STS designs lack simultaneous multimodal perception and suffer from unreliable tactile tracking. Furthermore, integrating these rich multimodal signals into learning-based manipulation pipelines remains an open challenge. We introduce TacThru, an STS sensor enabling simultaneous visual perception and robust tactile signal extraction, and TacThru-UMI, an imitation learning framework that leverages these multimodal signals for manipulation. Our sensor features a fully transparent elastomer, persistent illumination, novel keyline markers, and efficient tracking, while our learning system integrates these signals through a Transformer-based Diffusion Policy. Experiments on five challenging real-world tasks show that TacThru-UMI achieves an average success rate of 85.5%, significantly outperforming the baselines of tactile policy(66.3%) and vision-only policy (55.4%). The system excels in critical scenarios, including contact detection with thin and soft objects and precision manipulation requiring multimodal coordination. This work demonstrates that combining simultaneous multimodal perception with modern learning frameworks enables more precise, adaptable robotic manipulation.