# VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation

arXiv: https://arxiv.org/abs/2608.21290 · HTML: https://mlsanity.com/paper/2608.21290/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 36 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 30 of 31)

- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [Sparsh: Self-supervised touch representations for vision-based tactile sensing](https://mlsanity.com/paper/2410.24090/) (2024)
- [Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing](https://mlsanity.com/paper/2312.01853/) (2023)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks](https://mlsanity.com/paper/2406.13640/) (2024)
- [OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing](https://mlsanity.com/paper/2508.08706/) (2025)
- [ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation](https://mlsanity.com/paper/2506.15953/) (2025)
- [OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2603.19201/) (2026)
- [ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface](https://mlsanity.com/paper/2504.06156/) (2025)
- [FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation](https://mlsanity.com/paper/2506.01941/) (2025)
- [Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation](https://mlsanity.com/paper/2606.08737/) (2026)
- [UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking](https://mlsanity.com/paper/2602.10093/) (2026)
- [VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies](https://mlsanity.com/paper/2403.11898/) (2024)
- [UniT: Data Efficient Tactile Representation with Generalization to Unseen Objects](https://mlsanity.com/paper/2408.06481/) (2024)
- [Touch begins where vision ends: Generalizable policies for contact-rich manipulation](https://mlsanity.com/paper/2506.13762/) (2025)
- [ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning](https://mlsanity.com/paper/2505.18472/) (2025)
- [FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation](https://mlsanity.com/paper/2606.13102/) (2026)
- [TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2606.11184/) (2026)
- [Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention](https://mlsanity.com/paper/2606.26663/) (2026)
- [Learn from Incomplete Tactile Data: Tactile Representation Learning with Masked Autoencoders](https://mlsanity.com/paper/2307.07358/) (2023)
- [FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation](https://mlsanity.com/paper/2606.08555/) (2026)
- [N0-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens](https://mlsanity.com/paper/2607.23782/) (2026)
- [UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation](https://mlsanity.com/paper/2606.31451/) (2026)
- [UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models](https://mlsanity.com/paper/2606.31723/) (2026)
- [ManiSkill-ViTac 2025: Challenge on Manipulation Skill Learning With Vision and Tactile Sensing](https://mlsanity.com/paper/2411.12503/) (2024)
- [TacO: Benchmarking Tactile Sensors for Object Manipulation](https://mlsanity.com/paper/2605.21976/) (2026)
- [TactiDex: A Real-World Tactile-Guided Benchmark for Human-Like Dexterous Manipulation](https://mlsanity.com/paper/2607.09190/) (2026)

## Abstract (arXiv)

We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.