# OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

arXiv: https://arxiv.org/abs/2603.19201 · HTML: https://mlsanity.com/paper/2603.19201/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 7 of its 63 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 7 of 7)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Octopi: Object Property Reasoning with Large Tactile-Language Models](https://mlsanity.com/paper/2405.02794/) (2024)
- [VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback](https://mlsanity.com/paper/2507.17294/) (2025)
- [OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing](https://mlsanity.com/paper/2508.08706/) (2025)
- [MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation](https://mlsanity.com/paper/2509.26642/) (2025)
- [Demonstrating the Octopi-1.5 Visual-Tactile-Language Model](https://mlsanity.com/paper/2507.09985/) (2025)

## Built on by (most cited first, 30 of 36)

- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2606.11184/) (2026)
- [VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2607.02503/) (2026)
- [WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform](https://mlsanity.com/paper/2605.17912/) (2026)
- [FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation](https://mlsanity.com/paper/2606.08555/) (2026)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models](https://mlsanity.com/paper/2606.31723/) (2026)
- [Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation](https://mlsanity.com/paper/2607.01067/) (2026)
- [ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation](https://mlsanity.com/paper/2607.22530/) (2026)
- [ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulation](https://mlsanity.com/paper/2606.13877/) (2026)
- [FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning](https://mlsanity.com/paper/2607.24267/) (2026)
- [TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Prediction](https://mlsanity.com/paper/2607.28391/) (2026)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation](https://mlsanity.com/paper/2606.11637/) (2026)
- [Inference-time Policy Steering via Vision and Touch](https://mlsanity.com/paper/2606.14981/) (2026)
- [SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)](https://mlsanity.com/paper/2607.04234/) (2026)
- [N0-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2607.23783/) (2026)
- [HiTac-WAM: A Hierarchical Tactile World Action Model for Contact-Rich Robot Manipulation](https://mlsanity.com/paper/2608.19574/) (2026)
- [TacForcing: Streaming Action Generation with Execution-Time Tactile Feedback](https://mlsanity.com/paper/2608.25798/) (2026)
- [TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation](https://mlsanity.com/paper/2609.05266/) (2026)
- [TACO: TActile World Model as a Self-COrrector for Scalable Robot Policy Post-Training](https://mlsanity.com/paper/2607.02840/) (2026)
- [Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning](https://mlsanity.com/paper/2608.07558/) (2026)
- [World Models for Embodied Intelligence: From Plausible to Controllable to Actionable](https://mlsanity.com/paper/2609.16697/) (2026)
- [Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance](https://mlsanity.com/paper/2605.14571/) (2026)
- [Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models](https://mlsanity.com/paper/2608.00547/) (2026)
- [VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation](https://mlsanity.com/paper/2608.21290/) (2026)
- [N0-Foundation: Towards the Age of Tactile Intelligence](https://mlsanity.com/paper/2608.29601/) (2026)
- [DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination](https://mlsanity.com/paper/2609.09119/) (2026)

## Abstract (arXiv)

Contact-rich manipulation tasks, such as wiping and assembly, require accurate perception of contact forces, friction changes, and state transitions that cannot be reliably inferred from vision alone. Despite growing interest in visuo-tactile manipulation, progress is constrained by two persistent limitations: existing datasets are small in scale and narrow in task coverage, and current methods treat tactile signals as passive observations rather than using them to model contact dynamics or enable closed-loop control explicitly. In this paper, we present OmniViTac, a large-scale visuo-tactile-action dataset comprising 21,000+ trajectories across 86 tasks and 100+ objects, organized into six physics-grounded interaction patterns. Building on this dataset, we propose OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules: a self-supervised tactile encoder, a two-stream visuo-tactile world model for predicting short-horizon contact evolution, a contact-aware fusion policy for action generation, and a 60Hz reflexive controller that corrects deviations between predicted and observed tactile signals in a closed loop. Real-robot experiments across all six interaction categories show that OmniVTA outperforms existing methods and generalizes well to unseen objects and geometric configurations, confirming the value of combining predictive contact modeling with high-frequency tactile feedback for contact-rich manipulation. All data, models, and code will be made publicly available on the project website at https://mrsecant.github.io/OmniVTA.