# villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2507.23682 · HTML: https://mlsanity.com/paper/2507.23682/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 58 of its 65 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)
- [Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations](https://mlsanity.com/paper/2412.14803/) (2024)

## Built on by (most cited first, 30 of 63)

- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [Being-H0.7: A Latent World-Action Model from Egocentric Videos](https://mlsanity.com/paper/2605.00078/) (2026)
- [Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey](https://mlsanity.com/paper/2508.13073/) (2025)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model](https://mlsanity.com/paper/2602.12063/) (2026)
- [VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models](https://mlsanity.com/paper/2512.22539/) (2025)
- [RLDX-1 Technical Report](https://mlsanity.com/paper/2605.03269/) (2026)
- [Latent Action Pretraining Through World Modeling](https://mlsanity.com/paper/2509.18428/) (2025)
- [PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2601.07060/) (2026)
- [DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA](https://mlsanity.com/paper/2603.29844/) (2026)
- [GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation](https://mlsanity.com/paper/2512.16811/) (2025)
- [FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model](https://mlsanity.com/paper/2603.10712/) (2026)
- [ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation](https://mlsanity.com/paper/2602.00557/) (2026)
- [Chain of World: World Model Thinking in Latent Motion](https://mlsanity.com/paper/2603.03195/) (2026)
- [LaMP: Learning Vision-Language-Action Policy with 3D Scene Flow as Latent Motion Prior](https://mlsanity.com/paper/2603.25399/) (2026)
- [Olaf-World: Orienting Latent Actions for Video World Modeling](https://mlsanity.com/paper/2602.10104/) (2026)
- [ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation](https://mlsanity.com/paper/2603.27670/) (2026)
- [MoTVLA: A Vision-Language-Action Model with Unified Fast-Slow Reasoning](https://mlsanity.com/paper/2510.18337/) (2025)
- [Learning to Act Robustly with View-Invariant Latent Actions](https://mlsanity.com/paper/2601.02994/) (2026)
- [RotVLA: Rotational Latent Action for Vision-Language-Action Model](https://mlsanity.com/paper/2605.13403/) (2026)
- [Point Tracking Improves World Action Models](https://mlsanity.com/paper/2605.23856/) (2026)
- [CollabVLA: Self-Reflective Vision-Language-Action Model Dreaming Together with Human](https://mlsanity.com/paper/2509.14889/) (2025)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models](https://mlsanity.com/paper/2602.20231/) (2026)
- [Veo-Act: Enhancing VLA Policies with Frontier Video Models](https://mlsanity.com/paper/2604.04502/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [LARA: Latent Action Representation Alignment for Vision-Language-Action Models](https://mlsanity.com/paper/2606.07100/) (2026)
- [From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models](https://mlsanity.com/paper/2605.04678/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.