mlsanityData to 14 Oct 2026

villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

arXiv 2507.23682 ↗First posted 31 Jul 2025cs.RO cs.AI cs.LGcited by 63

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 58 of its 65 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

58 of 65 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI17 Oct
  2. 2023Learning to Act without Actions17 Dec
  3. 2023Structured World Models from Human Videos21 Aug
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  10. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  11. 2024Genie: Generative Interactive Environmentscited by 747
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
38 more
  1. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  2. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  3. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  6. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  7. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  8. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  9. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  10. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  11. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  12. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 349
  13. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  14. 2024Latent Action Pretraining from Videoscited by 321
  15. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  16. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  17. 2022Rectified Flow: A Marginal Preserving Approach to Optimal Transportcited by 283
  18. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  19. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  20. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  21. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  22. 2020Spatial-Temporal Transformer Networks for Traffic Flow Forecastingcited by 180
  23. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  24. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  25. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  26. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  27. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  28. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  29. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  30. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 89
  31. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  32. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 80
  33. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  34. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  35. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 44
  36. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 41
  37. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  38. 2025What Do Latent Action Models Actually Learn?cited by 32

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 53.

63 papers.

Sort:

By area:

  1. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  2. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  3. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  5. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  6. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
  7. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 41
  8. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 31
  9. 2026RLDX-1 Technical Reportcited by 28
  10. 2025Latent Action Pretraining Through World Modelingcited by 24
  11. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  12. 2026DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLAcited by 19
38 more
  1. 2025GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulationcited by 17
  2. 2026FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Modelcited by 16
  3. 2026ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulationcited by 14
  4. 2026Chain of World: World Model Thinking in Latent Motioncited by 11
  5. 2026LaMP: Learning Vision-Language-Action Policy with 3D Scene Flow as Latent Motion Priorcited by 11
  6. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 10
  7. 2026ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulationcited by 10
  8. 2025MoTVLA: A Vision-Language-Action Model with Unified Fast-Slow Reasoningcited by 9
  9. 2026Learning to Act Robustly with View-Invariant Latent Actionscited by 8
  10. 2026RotVLA: Rotational Latent Action for Vision-Language-Action Modelcited by 8
  11. 2026Point Tracking Improves World Action Modelscited by 7
  12. 2025CollabVLA: Self-Reflective Vision-Language-Action Model Dreaming Together with Humancited by 6
  13. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 6
  14. 2026UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Modelscited by 5
  15. 2026Veo-Act: Enhancing VLA Policies with Frontier Video Modelscited by 5
  16. 2026Robot Learning from Human Videos: A Surveycited by 5
  17. 2026LARA: Latent Action Representation Alignment for Vision-Language-Action Modelscited by 5
  18. 2026From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Modelscited by 4
  19. 2026Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptationcited by 4
  20. 2026Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Actioncited by 4
  21. 2026SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Spacecited by 4
  22. 2026What Matters for Latent Actions in Robot Learningcited by 4
  23. 2025Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectoriescited by 3
  24. 2026Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Modelscited by 3
  25. 2026From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Datacited by 3
  26. 2026Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Controlcited by 3
  27. 2025TrajBooster: Boosting Humanoid Whole-Body Manipulation via Trajectory-Centric Learningcited by 2
  28. 2025UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learningcited by 2
  29. 2026Why Latent Actions Fail, and How to Prevent Itcited by 2
  30. 2026Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Modelscited by 2
  31. 2026ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadowcited by 2
  32. 2026Beyond Dense Futures: World Models as Structured Planners for Robotic Manipulationcited by 1
  33. 2026Reinforcing VLAs in Task-Agnostic World Modelscited by 1
  34. 2026SCAR: Self-Supervised Continuous Action Representation Learningcited by 1
  35. 2026FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learningcited by 1
  36. 2026From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligencecited by 1
  37. 2026CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulatorscited by 1
  38. 2026Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervisioncited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LoLA: Long Horizon Latent Action Learning for General Robot Manipulation
  2. 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models
  3. 2026Cross-Hand Latent Representation for Vision-Language-Action Models
  4. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  5. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  6. 2026SLIP-VLA: Single-Step Latent Imagination for Policy Learning in Vision-Language-Action Models
  7. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
  8. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models