# GaussianAD: Gaussian-Centric End-to-End Autonomous Driving

arXiv: https://arxiv.org/abs/2412.10371 · HTML: https://mlsanity.com/paper/2412.10371/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 24 of its 66 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Placed in

- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 24 of 24)

- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2208.14437/) (2022)
- [VectorMapNet: End-to-end Vectorized HD Map Learning](https://mlsanity.com/paper/2206.08920/) (2022)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2405.17429/) (2024)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction](https://mlsanity.com/paper/2412.04384/) (2024)
- [FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving](https://mlsanity.com/paper/2308.01006/) (2023)
- [PointOcc: Cylindrical Tri-Perspective View for Point-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2308.16896/) (2023)
- [SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving](https://mlsanity.com/paper/2404.06892/) (2024)
- [Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction](https://mlsanity.com/paper/2212.02181/) (2022)
- [Rethinking Imitation-based Planner for Autonomous Driving](https://mlsanity.com/paper/2309.10443/) (2023)
- [Hierarchical Generative Adversarial Imitation Learning with Mid-level Input Generation for Autonomous Driving on Urban Environments](https://mlsanity.com/paper/2302.04823/) (2023)
- [CCIL: Context-conditioned imitation learning for urban driving](https://mlsanity.com/paper/2305.02649/) (2023)
- [GPD-1: Generative Pre-training for Driving](https://mlsanity.com/paper/2412.08643/) (2024)

## Built on by (most cited first, 30 of 31)

- [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flow](https://mlsanity.com/paper/2502.17288/) (2025)
- [DVGT: Driving Visual Geometry Transformer](https://mlsanity.com/paper/2512.16919/) (2025)
- [QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.10977/) (2025)
- [GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.00034/) (2025)
- [PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.17596/) (2025)
- [InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.13047/) (2025)
- [GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention](https://mlsanity.com/paper/2505.10685/) (2025)
- [DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale](https://mlsanity.com/paper/2604.00813/) (2026)
- [MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.04441/) (2025)
- [GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2507.19451/) (2025)
- [Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving](https://mlsanity.com/paper/2603.27287/) (2026)
- [MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving](https://mlsanity.com/paper/2602.21952/) (2026)
- [Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2503.14537/) (2025)
- [Vega: Learning to Drive with Natural Language Instructions](https://mlsanity.com/paper/2603.25741/) (2026)
- [SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving](https://mlsanity.com/paper/2509.16588/) (2025)
- [WPT: World-to-Policy Transfer via Online World Model Distillation](https://mlsanity.com/paper/2511.20095/) (2025)
- [SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model](https://mlsanity.com/paper/2511.22039/) (2025)
- [RadarGaussianDet3D: Gaussian Representation-based Real-time 3D Object Detection with 4D Automotive Radars](https://mlsanity.com/paper/2509.16119/) (2025)
- [Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models](https://mlsanity.com/paper/2510.16729/) (2025)
- [ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding](https://mlsanity.com/paper/2512.03370/) (2025)
- [UniGS: Unified Geometry-Aware Gaussian Splatting for Multimodal Rendering](https://mlsanity.com/paper/2510.12174/) (2025)
- [DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving](https://mlsanity.com/paper/2604.00969/) (2026)
- [TGP: Two-modal occupancy prediction with 3D Gaussian and sparse points for 3D Environment Awareness](https://mlsanity.com/paper/2503.09941/) (2025)
- [S2GO: Streaming Sparse Gaussian Occupancy Prediction](https://mlsanity.com/paper/2506.05473/) (2025)
- [SuperQuadricOcc: Real-Time Self-Supervised Semantic Occupancy Estimation with Superquadric Volume Rendering](https://mlsanity.com/paper/2511.17361/) (2025)
- [GaussianMap: Learning Gaussian Representation for Multi-Sensor Online HD Map Construction](https://mlsanity.com/paper/2606.31177/) (2026)
- [OPUS-V2: Bridging the Gap between Sparse Points and Dense Voxels](https://mlsanity.com/paper/2608.29187/) (2026)

## Abstract (arXiv)

Vision-based autonomous driving shows great potential due to its satisfactory performance and low costs. Most existing methods adopt dense representations (e.g., bird's eye view) or sparse representations (e.g., instance boxes) for decision-making, which suffer from the trade-off between comprehensiveness and efficiency. This paper explores a Gaussian-centric end-to-end autonomous driving (GaussianAD) framework and exploits 3D semantic Gaussians to extensively yet sparsely describe the scene. We initialize the scene with uniform 3D Gaussians and use surrounding-view images to progressively refine them to obtain the 3D Gaussian scene representation. We then use sparse convolutions to efficiently perform 3D perception (e.g., 3D detection, semantic map construction). We predict 3D flows for the Gaussians with dynamic semantics and plan the ego trajectory accordingly with an objective of future scene forecasting. Our GaussianAD can be trained in an end-to-end manner with optional perception labels when available. Extensive experiments on the widely used nuScenes dataset verify the effectiveness of our end-to-end GaussianAD on various tasks including motion planning, 3D occupancy prediction, and 4D occupancy forecasting. Code: https://github.com/wzzheng/GaussianAD.