mlsanityData to 14 Oct 2026

FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving

arXiv 2505.17685 ↗First posted 23 May 2025cs.CVcited by 166

On the map: Vision-language-action models → VLA for autonomous driving

also in 1 more
Abstract (arXiv)

Vision-Language-Action (VLA) models offer significant potential for end-to-end driving, yet their reasoning is often constrained by textual Chains-of-Thought (CoT). This symbolic compression of visual information creates a modality gap between perception and planning by blurring spatio-temporal relations and discarding fine-grained cues. We introduce FSDrive, a framework that empowers VLAs to "think visually" using a novel visual spatio-temporal CoT. FSDrive first operates as a world model, generating a unified future frame that combines a predicted background with explicit, physically-plausible priors like future lane dividers and 3D object boxes. This imagined scene serves as the visual spatio-temporal CoT, capturing both spatial structure and temporal evolution in a single representation. The same VLA then functions as an inverse-dynamics model to plan trajectories conditioned on current observations and this visual CoT. We enable this with a unified pre-training paradigm that expands the model's vocabulary with visual tokens and jointly optimizes for semantic understanding (VQA) and future-frame prediction. A progressive curriculum first generates structural priors to enforce physical laws before rendering the full scene. Evaluations on nuScenes and NAVSIM show FSDrive improves trajectory accuracy and reduces collisions, while also achieving competitive FID for video generation with a lightweight autoregressive model and advancing scene understanding on DriveLM. These results confirm that our visual spatio-temporal CoT bridges the perception-planning gap, enabling safer, more anticipatory autonomous driving. Code is available at https://github.com/MIV-XJTU/FSDrive.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Placed in VLA for autonomous driving, one of 13 directions in Vision-language-action models. Builds on 82 of its 114 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

82 of 114 references.

References · 82

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2022Planning-oriented Autonomous Drivingcited by 1,087
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  12. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
38 more
  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  4. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  6. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  8. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  9. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  10. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  11. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  12. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  13. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  14. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  15. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  16. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  17. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  18. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  19. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  20. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  21. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  22. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  23. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  24. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
  25. 2023A Language Agent for Autonomous Drivingcited by 164
  26. 2025Do generative video models understand physical principles?cited by 161
  27. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  28. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  29. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  30. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  31. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  32. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  33. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  34. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  35. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 123
  36. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  37. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 106
  38. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 99

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 66; 2026: 100.

166 papers.

Sort:

By area:

  1. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  2. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 134
  3. 2026ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learningcited by 103
  4. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  5. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  6. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  8. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  9. 2025World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Trainingcited by 59
  10. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  11. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  12. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 47
38 more
  1. 2025OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrievalcited by 40
  2. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 38
  3. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 37
  4. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 33
  5. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  6. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  7. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learningcited by 31
  8. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 28
  9. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 26
  10. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  11. 2025Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMAcited by 23
  12. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 23
  13. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
  14. 2025Evaluating Supervised Learning Models for Fraud Detection: A Comparative Study of Classical and Deep Architectures on Imbalanced Transaction Datacited by 20
  15. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  16. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
  17. 2025MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusioncited by 19
  18. 2025PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimizationcited by 19
  19. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 19
  20. 2026Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrievalcited by 18
  21. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  22. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 17
  23. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 17
  24. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 16
  25. 2025Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoningcited by 15
  26. 2025AMAS: Adaptively Determining Communication Topology for LLM-based Multi-Agent Systemcited by 15
  27. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  28. 2025Latent Chain-of-Thought World Modeling for End-to-End Drivingcited by 15
  29. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 15
  30. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  31. 2026HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrievalcited by 14
  32. 2026Demystifying Video Reasoningcited by 14
  33. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 14
  34. 2025AdsQA: Towards Advertisement Video Understandingcited by 13
  35. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  36. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 12
  37. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 12
  38. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026X-Mind: Efficient Visual Chain-of-Thought via Predictive World Model for End-to-End Driving
  2. 2025CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
  3. 2026Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving
  4. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions
  5. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving
  6. 2026ReDrive: Shaping Representations with World Modeling for End-to-End Driving
  7. 2026BrainWAM: Action-Space Coordination of Semantic Priors and Predictive Dynamics for Autonomous Driving
  8. 2025Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution