mlsanityData to 14 Oct 2026

VITAL: From Visual Motion to Precise Action

arXiv 2610.20604 ↗First posted 14 Oct 2026cs.CVcited by 0

On the map: World models → Embodied and robot world models also fits: World models for autonomous driving

Abstract (arXiv)

World-action models aim to turn visual anticipation into driving decisions, yet typically train action experts on trajectory representations defined independently of visual dynamics. This motivates deriving action targets directly from visual motion. We first examine a simpler setting in which trajectories are reconstructed from recorded future images. Even without the need for future prediction, reconstruction captures the dominant maneuver but leaves centimeter-scale errors. We therefore introduce VITAL (Vision-Induced Tokens for Action Learning), an action tokenizer that couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold. This representation makes visual motion the principal prediction target and gives small metric corrections a separate, normalized learning scale. Using flow matching, the action expert jointly predicts both components from the current image, ego state, and navigation command, without generating future images at inference. The same residual coordinates support bounded reinforcement post-training that refines execution while keeping the predicted visual scaffold fixed. With only one action denoising step, VITAL achieves 90.8 PDMS on NAVSIM v1 navtest, within 0.13 points of its ten-step score at 3.29x the end-to-end inference speed. Without retraining, it achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard. Project page: https://worldbench.github.io/vital

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 109 of its 117 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

109 of 117 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizers11 Jun
  2. 2026Learning Visual Feature-Based World Models via Residual Latent Action8 May
  3. 2026World Guidance: World Modeling in Condition Space for Action Generation25 Feb
  4. 2026World Action Models are Zero-shot Policies17 Feb
  5. 2025WorldVLA: Towards Autoregressive Action World Model26 Jun
  6. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning11 Jun
  7. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 101

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2023Consistency Modelscited by 1,822
  5. 2024Genie: Generative Interactive Environmentscited by 747
  6. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  7. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  8. 2025Mean Flows for One-step Generative Modelingcited by 561
  9. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  10. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  11. 2024One Step Diffusion via Shortcut Modelscited by 391
  12. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
38 more
  1. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  2. 2024Latent Action Pretraining from Videoscited by 321
  3. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  4. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  5. 2024Diffusion Policy Policy Optimizationcited by 280
  6. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  7. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  8. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
  9. 2025Flow Q-Learningcited by 184
  10. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  11. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  12. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 166
  13. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
  14. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  15. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  16. 2025Pseudo-Simulation for Autonomous Drivingcited by 150
  17. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 134
  18. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 129
  19. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
  20. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  21. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  22. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 104
  23. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 97
  24. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 96
  25. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
  26. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 94
  27. 2022LaserMix for Semi-Supervised LiDAR Semantic Segmentationcited by 93
  28. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  29. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 89
  30. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  31. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  32. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  33. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
  34. 20253D and 4D World Modeling: A Surveycited by 67
  35. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64
  36. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54
  37. 2024QueST: Self-Supervised Skill Abstractions for Learning Continuous Controlcited by 53
  38. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49

50 most cited of 101 All 101 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Native Action-Prior Learning from Videos for World Action Models
  2. 2026Grounding Driving VLA via Inverse Kinematics
  3. 2026WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving
  4. 2026An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
  5. 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
  6. 2026MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies
  7. 2026ForgeDrive: Bidirectional Cross-Conditioning for Unified Visual-Action Generation in Autonomous Driving
  8. 2026CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight