# DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale arXiv: https://arxiv.org/abs/2604.00813 · HTML: https://mlsanity.com/paper/2604.00813/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. - Builds on 84 of its 93 references that are in the map. - First posted on arXiv 1 Apr 2026. ## Placed in - End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/) ## Builds on (most cited first, 30 of 84) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021) - [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023) - [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025) - [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022) - [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024) - [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022) - [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021) - [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021) - [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023) - [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022) - [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025) - [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022) - [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023) - [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019) - [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023) - [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022) - [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024) - [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023) - [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022) - [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024) - [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025) - [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023) - [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025) - [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023) - [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024) - [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020) ## Built on by (most cited first, 13 of 13) - [Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation](https://mlsanity.com/paper/2604.18486/) (2026) - [SM4RT: Learning Structured Motion Geometry for 4D Reconstruction](https://mlsanity.com/paper/2607.22534/) (2026) - [VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving](https://mlsanity.com/paper/2606.12396/) (2026) - [GeoWAM: Visual Geometry World Action Models for Autonomous Driving](https://mlsanity.com/paper/2608.23486/) (2026) - [GeoWorldAD: Geometry World Action Model for Autonomous Driving](https://mlsanity.com/paper/2607.17521/) (2026) - [IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation](https://mlsanity.com/paper/2605.16258/) (2026) - [VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction](https://mlsanity.com/paper/2607.18078/) (2026) - [LiAuto-GeoX: Efficient Grounded Driving Transformer](https://mlsanity.com/paper/2606.05774/) (2026) - [Open-vocabulary 3D object detection with promptable segmentation](https://mlsanity.com/paper/2609.19358/) (2026) - [PhysWAM: Physically Consistent World Action Model for Autonomous Driving](https://mlsanity.com/paper/2609.37970/) (2026) - [Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?](https://mlsanity.com/paper/2610.09695/) (2026) - [VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving](https://mlsanity.com/paper/2610.11161/) (2026) - [Drive3R: Block-wise Streaming Geometric Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2610.15112/) (2026) ## Abstract (arXiv) End-to-end autonomous driving has evolved from the conventional paradigm based on sparse perception into vision-language-action (VLA) models, which focus on learning language descriptions as an auxiliary task to facilitate planning. In this paper, we propose an alternative Vision-Geometry-Action (VGA) paradigm that advocates dense 3D geometry as the critical cue for autonomous driving. As vehicles operate in a 3D world, we think dense 3D geometry provides the most comprehensive information for decision-making. However, most existing geometry reconstruction methods (e.g., DVGT) rely on computationally expensive batch processing of multi-frame inputs and cannot be applied to online planning. To address this, we introduce a streaming Driving Visual Geometry Transformer (DVGT-2), which processes inputs in an online manner and jointly outputs dense geometry and trajectory planning for the current frame. We employ temporal causal attention and cache historical features to support on-the-fly inference. To further enhance efficiency, we propose a sliding-window streaming strategy and use historical caches within a certain interval to avoid repetitive computations. Despite the faster speed, DVGT-2 achieves superior geometry reconstruction performance on various datasets. The same trained DVGT-2 can be directly applied to planning across diverse camera configurations without fine-tuning, including closed-loop NAVSIM and open-loop nuScenes benchmarks.