mlsanityData to 7 Oct 2026

Teaching Vision-Language-Action Models What to See and Where to Look

arXiv 2607.01658 ↗First posted 2 Jul 2026cs.CVcited by 1

On the map: Vision-language-action models → VLA for autonomous driving

also in 1 more
Abstract (arXiv)

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLA for autonomous driving, one of 13 directions in Vision-language-action models. Builds on 50 of its 56 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

50 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving14 Oct
  2. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision6 Sep
  3. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning16 Jun
  4. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models29 May
  5. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving22 May
  6. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model30 Mar

Other references · 44

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2022Planning-oriented Autonomous Drivingcited by 1,068
  9. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  10. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
32 more
  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  4. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  5. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  6. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  7. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  8. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 244
  9. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  10. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  11. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  12. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  13. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  14. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  15. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  16. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  17. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  18. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 93
  19. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  20. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 66
  21. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  22. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  23. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 41
  24. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  25. 2026Driving on Registerscited by 38
  26. 2023QCNeXt: A Next-Generation Framework For Joint Multi-Agent Trajectory Predictioncited by 31
  27. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  28. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  29. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 14
  30. 2025EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Drivingcited by 12
  31. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  32. 2026ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Drivingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026RefineDrive: Reliable Failure-Guided Learning for Vision-Language-Action Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
  2. 2026RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
  3. 2026Learning Vision-Language-Action World Models for Autonomous Driving
  4. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
  5. 2026DriveStack-VLA: Render-Teacher Alignment for BEV-Based DeepStack Vision-Language-Action Model
  6. 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving
  7. 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Driving