# ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation

arXiv: https://arxiv.org/abs/2506.15953 · HTML: https://mlsanity.com/paper/2506.15953/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2025.

## Built on by (most cited first, 30 of 38)

- [Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation](https://mlsanity.com/paper/2606.08737/) (2026)
- [Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation](https://mlsanity.com/paper/2512.23864/) (2025)
- [T-Rex: Tactile-Reactive Dexterous Manipulation](https://mlsanity.com/paper/2606.17055/) (2026)
- [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026)
- [Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation](https://mlsanity.com/paper/2512.09851/) (2025)
- [Multimodal Diffusion Forcing for Forceful Manipulation](https://mlsanity.com/paper/2511.04812/) (2025)
- [Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy](https://mlsanity.com/paper/2602.01939/) (2026)
- [DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter](https://mlsanity.com/paper/2602.05513/) (2026)
- [Tactile Modality Fusion for Vision-Language-Action Models](https://mlsanity.com/paper/2603.14604/) (2026)
- [ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulation](https://mlsanity.com/paper/2606.13877/) (2026)
- [TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Prediction](https://mlsanity.com/paper/2607.28391/) (2026)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [Inference-time Policy Steering via Vision and Touch](https://mlsanity.com/paper/2606.14981/) (2026)
- [N0-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2607.23783/) (2026)
- [HiTac-WAM: A Hierarchical Tactile World Action Model for Contact-Rich Robot Manipulation](https://mlsanity.com/paper/2608.19574/) (2026)
- [TacForcing: Streaming Action Generation with Execution-Time Tactile Feedback](https://mlsanity.com/paper/2608.25798/) (2026)
- [TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation](https://mlsanity.com/paper/2609.05266/) (2026)
- [ForeDiffusion: Foresight-Conditioned Diffusion Policy via Future View Construction for Robot Manipulation](https://mlsanity.com/paper/2601.12925/) (2026)
- [DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation](https://mlsanity.com/paper/2605.05241/) (2026)
- [Tac-DINO: Learning Vision-Tactile Features with Patch Alignment](https://mlsanity.com/paper/2606.12069/) (2026)
- [Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning](https://mlsanity.com/paper/2608.07558/) (2026)
- [Symmetry-Aware Fusion of Vision and Tactile Sensing via Bilateral Force Priors for Robotic Manipulation](https://mlsanity.com/paper/2602.13689/) (2026)
- [Cross-Modal Visuo-Tactile Object Perception](https://mlsanity.com/paper/2604.02108/) (2026)
- [Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation](https://mlsanity.com/paper/2605.28812/) (2026)
- [NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation](https://mlsanity.com/paper/2606.24450/) (2026)
- [Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs](https://mlsanity.com/paper/2607.00302/) (2026)
- [Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models](https://mlsanity.com/paper/2608.00547/) (2026)
- [VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation](https://mlsanity.com/paper/2608.21290/) (2026)
- [WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation](https://mlsanity.com/paper/2609.07002/) (2026)
- [DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination](https://mlsanity.com/paper/2609.09119/) (2026)

## Abstract (arXiv)

Dexterous manipulation is a cornerstone capability for robotic systems aiming to interact with the physical world in a human-like manner. Although vision-based methods have advanced rapidly, tactile sensing remains crucial for fine-grained control, particularly in unstructured or visually occluded settings. We present ViTacFormer, a representation-learning approach that couples a cross-attention encoder to fuse high-resolution vision and touch with an autoregressive tactile prediction head that anticipates future contact signals. Building on this architecture, we devise an easy-to-challenging curriculum that steadily refines the visual-tactile latent space, boosting both accuracy and robustness. The learned cross-modal representation drives imitation learning for multi-fingered hands, enabling precise and adaptive manipulation. Across a suite of challenging real-world benchmarks, our method achieves approximately 50% higher success rates than prior state-of-the-art systems. To our knowledge, it is also the first to autonomously complete long-horizon dexterous manipulation tasks that demand highly precise control with an anthropomorphic hand, successfully executing up to 11 sequential stages and sustaining continuous operation for 2.5 minutes.