# textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

arXiv: https://arxiv.org/abs/2405.20323 · HTML: https://mlsanity.com/paper/2405.20323/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 58 of its 68 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)

## Builds on (most cited first, 30 of 58)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)

## Built on by (most cited first, 30 of 105)

- [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025)
- [DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation](https://mlsanity.com/paper/2410.13571/) (2024)
- [Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillation](https://mlsanity.com/paper/2503.12820/) (2025)
- [ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration](https://mlsanity.com/paper/2411.19548/) (2024)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes](https://mlsanity.com/paper/2501.00602/) (2024)
- [GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splatting](https://mlsanity.com/paper/2408.11447/) (2024)
- [DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes](https://mlsanity.com/paper/2411.11921/) (2024)
- [ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation](https://mlsanity.com/paper/2503.18438/) (2025)
- [DreamDrive: Generative 4D Scene Modeling from Street View Images](https://mlsanity.com/paper/2501.00601/) (2024)
- [Segment Any Motion in Videos](https://mlsanity.com/paper/2503.22268/) (2025)
- [FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes](https://mlsanity.com/paper/2412.03566/) (2024)
- [DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving](https://mlsanity.com/paper/2412.09043/) (2024)
- [CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving](https://mlsanity.com/paper/2503.06744/) (2025)
- [PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views](https://mlsanity.com/paper/2410.18979/) (2024)
- [Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model](https://mlsanity.com/paper/2412.05280/) (2024)
- [4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives](https://mlsanity.com/paper/2412.20720/) (2024)
- [WAM4D: Fast 4D World Action Model via Spatial Register Tokens](https://mlsanity.com/paper/2606.14048/) (2026)
- [Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model](https://mlsanity.com/paper/2412.09647/) (2024)
- [Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks](https://mlsanity.com/paper/2510.19195/) (2025)
- [EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splatting](https://mlsanity.com/paper/2411.15582/) (2024)
- [WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving](https://mlsanity.com/paper/2509.23402/) (2025)
- [VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model](https://mlsanity.com/paper/2603.12655/) (2026)
- [ManipDreamer3D : Synthesizing Plausible Robotic Manipulation Video with Occupancy-aware 3D Trajectory](https://mlsanity.com/paper/2509.05314/) (2025)
- [CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Driving](https://mlsanity.com/paper/2605.10426/) (2026)
- [Driving View Synthesis on Free-form Trajectories with Generative Prior](https://mlsanity.com/paper/2412.01717/) (2024)
- [ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance](https://mlsanity.com/paper/2504.16464/) (2025)
- [CRUISE: Cooperative Reconstruction and Editing in V2X Scenarios using Gaussian Splatting](https://mlsanity.com/paper/2507.18473/) (2025)
- [UniSplat: Unified Spatio-Temporal Fusion via 3D Latent Scaffolds for Dynamic Driving Scene Reconstruction](https://mlsanity.com/paper/2511.04595/) (2025)
- [GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting](https://mlsanity.com/paper/2411.12981/) (2024)

## Abstract (arXiv)

Photorealistic 3D reconstruction of street scenes is a critical technique for developing real-world simulators for autonomous driving. Despite the efficacy of Neural Radiance Fields (NeRF) for driving scenes, 3D Gaussian Splatting (3DGS) emerges as a promising direction due to its faster speed and more explicit representation. However, most existing street 3DGS methods require tracked 3D vehicle bounding boxes to decompose the static and dynamic elements for effective reconstruction, limiting their applications for in-the-wild scenarios. To facilitate efficient 3D scene reconstruction without costly annotations, we propose a self-supervised street Gaussian (textitS3Gaussian) method to decompose dynamic and static elements from 4D consistency. We represent each scene with 3D Gaussians to preserve the explicitness and further accompany them with a spatial-temporal field network to compactly model the 4D dynamics. We conduct extensive experiments on the challenging Waymo-Open dataset to evaluate the effectiveness of our method. Our textitS3Gaussian demonstrates the ability to decompose static and dynamic scenes and achieves the best performance without using 3D annotations. Code is available at: https://github.com/nnanhuang/S3Gaussian/.