# VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs

arXiv: https://arxiv.org/abs/2603.23481 · HTML: https://mlsanity.com/paper/2603.23481/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 41 of its 48 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge](https://mlsanity.com/paper/2507.04447/) (2025)
- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation](https://mlsanity.com/paper/2508.05635/) (2025)
- [Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation](https://mlsanity.com/paper/2503.02881/) (2025)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation](https://mlsanity.com/paper/2505.22159/) (2025)
- [3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing](https://mlsanity.com/paper/2410.24091/) (2024)
- [GeoVLA: Empowering 3D Representations in Vision-Language-Action Models](https://mlsanity.com/paper/2508.09071/) (2025)
- [Sparsh: Self-supervised touch representations for vision-based tactile sensing](https://mlsanity.com/paper/2410.24090/) (2024)
- [VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation](https://mlsanity.com/paper/2505.09577/) (2025)
- [TLA: Tactile-Language-Action Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2503.08548/) (2025)
- [VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback](https://mlsanity.com/paper/2507.17294/) (2025)

## Built on by (most cited first, 17 of 17)

- [T-Rex: Tactile-Reactive Dexterous Manipulation](https://mlsanity.com/paper/2606.17055/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform](https://mlsanity.com/paper/2605.17912/) (2026)
- [FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulation](https://mlsanity.com/paper/2607.28596/) (2026)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [Inference-time Policy Steering via Vision and Touch](https://mlsanity.com/paper/2606.14981/) (2026)
- [N0-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2607.23783/) (2026)
- [τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision](https://mlsanity.com/paper/2607.24485/) (2026)
- [TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation](https://mlsanity.com/paper/2609.05266/) (2026)
- [Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning](https://mlsanity.com/paper/2608.07558/) (2026)
- [A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications](https://mlsanity.com/paper/2608.20379/) (2026)
- [Agile-WAM: An Agile Tactile World Action Model for Contact-Rich Robot Control](https://mlsanity.com/paper/2609.20761/) (2026)
- [ProxiDex: Learning Dynamics-Guided Proximity Policy for Dexterous Manipulation](https://mlsanity.com/paper/2609.16586/) (2026)
- [DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation](https://mlsanity.com/paper/2609.24976/) (2026)
- [Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation](https://mlsanity.com/paper/2609.34182/) (2026)
- [TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model](https://mlsanity.com/paper/2610.00638/) (2026)
- [DexFLEX: Contact-Aware Foundation Controller for Command-Guided Dexterity](https://mlsanity.com/paper/2610.14372/) (2026)

## Abstract (arXiv)

Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90 percent on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.