VITAL: From Visual Motion to Precise Action
On the map: World models → Embodied and robot world models also fits: World models for autonomous driving
Abstract (arXiv)
World-action models aim to turn visual anticipation into driving decisions, yet typically train action experts on trajectory representations defined independently of visual dynamics. This motivates deriving action targets directly from visual motion. We first examine a simpler setting in which trajectories are reconstructed from recorded future images. Even without the need for future prediction, reconstruction captures the dominant maneuver but leaves centimeter-scale errors. We therefore introduce VITAL (Vision-Induced Tokens for Action Learning), an action tokenizer that couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold. This representation makes visual motion the principal prediction target and gives small metric corrections a separate, normalized learning scale. Using flow matching, the action expert jointly predicts both components from the current image, ego state, and navigation command, without generating future images at inference. The same residual coordinates support bounded reinforcement post-training that refines execution while keeping the predicted visual scaffold fixed. With only one action denoising step, VITAL achieves 90.8 PDMS on NAVSIM v1 navtest, within 0.13 points of its ten-step score at 3.29x the end-to-end inference speed. Without retraining, it achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard. Project page: https://worldbench.github.io/vital
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 109 of its 117 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
109 of 117 references.
Closest ancestors · 8
Same direction, newest first
- 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizers11 Jun
- 2026Learning Visual Feature-Based World Models via Residual Latent Action8 May
- 2026World Guidance: World Modeling in Condition Space for Action Generation25 Feb
- 2026World Action Models are Zero-shot Policies17 Feb
- 2025WorldVLA: Towards Autoregressive Action World Model26 Jun
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning11 Jun
- 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
Other references · 101
- 2017Attention Is All You Needcited by 60,361
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
- 2023Consistency Modelscited by 1,822
- 2024Genie: Generative Interactive Environmentscited by 747
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
- 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
- 2025Mean Flows for One-step Generative Modelingcited by 561
- 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
- 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
- 2024One Step Diffusion via Shortcut Modelscited by 391
- 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
38 more
- 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
- 2024Latent Action Pretraining from Videoscited by 321
- 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
- 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
- 2024Diffusion Policy Policy Optimizationcited by 280
- 2018Residual Reinforcement Learning for Robot Controlcited by 274
- 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
- 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
- 2025Flow Q-Learningcited by 184
- 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
- 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
- 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 166
- 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
- 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
- 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
- 2025Pseudo-Simulation for Autonomous Drivingcited by 150
- 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 134
- 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 129
- 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
- 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
- 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
- 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 104
- 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 97
- 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 96
- 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
- 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 94
- 2022LaserMix for Semi-Supervised LiDAR Semantic Segmentationcited by 93
- 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
- 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 89
- 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
- 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
- 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
- 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
- 20253D and 4D World Modeling: A Surveycited by 67
- 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64
- 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54
- 2024QueST: Self-Supervised Skill Abstractions for Learning Continuous Controlcited by 53
- 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
50 most cited of 101 All 101 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Native Action-Prior Learning from Videos for World Action Models
- 2026Grounding Driving VLA via Inverse Kinematics
- 2026WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving
- 2026An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
- 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
- 2026MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies
- 2026ForgeDrive: Bidirectional Cross-Conditioning for Unified Visual-Action Generation in Autonomous Driving
- 2026CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight