ST4VLA: Spatially Guided Training for Vision-Language-Action Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Large vision-language models (VLMs) excel at multimodal understanding but fall short when extended to embodied tasks, where instructions must be transformed into low-level motor actions. We introduce ST4VLA, a dual-system Vision-Language-Action framework that leverages Spatial Guided Training to align action learning with spatial priors in VLMs. ST4VLA includes two stages: (i) spatial grounding pre-training, which equips the VLM with transferable priors via scalable point, box, and trajectory prediction from both web-scale and robot-specific data, and (ii) spatially guided action post-training, which encourages the model to produce richer spatial priors to guide action generation via spatial prompting. This design preserves spatial grounding during policy learning and promotes consistent optimization across spatial and action objectives. Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 -> 84.6 on Google Robot and from 54.7 -> 73.2 on WidowX Robot, establishing new state-of-the-art results on SimplerEnv. It also demonstrates stronger generalization to unseen objects and paraphrased instructions, as well as robustness to long-horizon perturbations in real-world settings. These results highlight scalable spatially guided training as a promising direction for robust, generalizable robot learning. Source code, data and models are released at https://internrobotics.github.io/internvla-m1.github.io/
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 84 of its 91 references that are in the map. First posted on arXiv 10 Feb 2026.
Builds on
84 of 91 references.
Closest ancestors · 12
Same direction, newest first
- 2025RoboBrain 2.0 Technical Report2 Jul
- 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
- 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics2 Jun
- 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces30 May
- 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better29 May
- 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models27 Mar
- 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
- 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
- 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policy28 Jun
- 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning17 Jun
- 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
Other references · 72
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
- 2016Modeling Context in Referring Expressionscited by 1,054
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
- 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
- 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
- 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
- 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
- 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
- 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
- 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
- 2024Latent Action Pretraining from Videoscited by 302
- 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
- 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
- 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
- 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
- 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
- 2023Any-point Trajectory Modeling for Policy Learningcited by 207
- 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 205
- 2024RT-H: Action Hierarchies Using Languagecited by 204
- 2025Unified Video Action Modelcited by 195
- 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
- 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
- 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
- 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
- 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
- 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
- 2025Unified Vision-Language-Action Modelcited by 128
- 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
- 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
- 2025GR-3 Technical Reportcited by 110
- 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
50 most cited of 72 All 72 (JSON)
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
- 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
- 2026FocusVLA: Focused Visual Utilization for Vision-Language-Action Modelscited by 5
- 2026Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulationcited by 3
- 2026QuoVLA: Quotient Space for Vision-Language-Action Modelscited by 2
- 2026VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Modelscited by 2
- 2026PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Controlcited by 0
- 2026PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control25 Aug 2026
- 2026VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models28 May 2026
- 2026QuoVLA: Quotient Space for Vision-Language-Action Models24 May 2026
- 2026Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation12 May 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond24 Apr 2026
- 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systems13 Apr 2026
- 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing6 Apr 2026
- 2026FocusVLA: Focused Visual Utilization for Vision-Language-Action Models30 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
- 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy
- 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
- 2026LA4VLA: Learning to Act without Seeing via Language-Action Pretraining
- 2026ST-π: Structured SpatioTemporal VLA for Robotic Manipulation
- 2025ROSA: Harnessing Robot States for Vision-Language and Action Alignment
- 2026EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
- 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey