mlsanityData to 7 Oct 2026

HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving

arXiv 2412.01718 ↗First posted 2 Dec 2024cs.CV cs.ROcited by 84

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

In the past few decades, autonomous driving algorithms have made significant progress in perception, planning, and control. However, evaluating individual components does not fully reflect the performance of entire systems, highlighting the need for more holistic assessment methods. This motivates the development of HUGSIM, a closed-loop, photo-realistic, and real-time simulator for evaluating autonomous driving algorithms. We achieve this by lifting captured 2D RGB images into the 3D space via 3D Gaussian Splatting, improving the rendering quality for closed-loop scenarios, and building the closed-loop environment. In terms of rendering, We tackle challenges of novel view synthesis in closed-loop scenarios, including viewpoint extrapolation and 360-degree vehicle rendering. Beyond novel view synthesis, HUGSIM further enables the full closed simulation loop, dynamically updating the ego and actor states and observations based on control commands. Moreover, HUGSIM offers a comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo, nuScenes, and PandaSet, along with over 400 varying scenarios, providing a fair and realistic evaluation platform for existing autonomous driving algorithms. HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks the potential for fine-tuning autonomous driving algorithms in a photorealistic closed-loop setting.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 53 of its 81 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

53 of 81 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024OmniRe: Omni Urban Scene Reconstruction29 Aug
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  3. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior29 Mar
  4. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  5. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting2 Jan
  6. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  7. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 46

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  6. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  7. 2022Planning-oriented Autonomous Drivingcited by 1,068
  8. 20242D Gaussian Splatting for Geometrically Accurate Radiance Fieldscited by 1,061
  9. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  10. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  11. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  12. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
34 more
  1. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  2. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  3. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  4. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  5. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  6. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  7. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  8. 2023GS-IR: 3D Gaussian Splatting for Inverse Renderingcited by 220
  9. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  10. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186
  11. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  12. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  13. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  14. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  15. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  16. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 141
  17. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  18. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  19. 2020Category Level Object Pose Estimation via Neural Analysis-by-Synthesiscited by 100
  20. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 95
  21. 2023GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorizationcited by 86
  22. 2021InverseForm: A Loss Function for Structured Boundary-Aware Segmentationcited by 65
  23. 2024NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Drivingcited by 61
  24. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55
  25. 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 52
  26. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  27. 2023Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervisioncited by 44
  28. 20243DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Viewscited by 37
  29. 2024GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Drivingcited by 17
  30. 2023LimSim: A Long-term Interactive Multi-scenario Traffic Simulatorcited by 13
  31. 2020Control Strategies for Autonomous Vehiclescited by 8
  32. 2024Efficient Depth-Guided Urban View Synthesiscited by 7
  33. 2024RoGs: Large Scale Road Surface Reconstruction with Meshgrid Gaussiancited by 5
  34. 2024DHGS: Decoupled Hybrid Gaussian Splatting for Driving Scenecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 31; 2026: 52.

84 papers.

Sort:

By area:

  1. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  2. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 76
  3. 20253D and 4D World Modeling: A Surveycited by 64
  4. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  5. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  6. 2026Driving on Registerscited by 38
  7. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  8. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  9. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  10. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  11. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  12. 2025LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Drivingcited by 22
38 more
  1. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 17
  2. 2025Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorercited by 16
  3. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  4. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
  5. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  6. 2026NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulationcited by 14
  7. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Drivingcited by 12
  8. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  9. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  10. 2026DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancercited by 10
  11. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  12. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  13. 2026BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Drivingcited by 9
  14. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  15. 2026RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Frameworkcited by 7
  16. 2026Test-Time Trajectory Optimization for Autonomous Drivingcited by 7
  17. 2026Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negativescited by 6
  18. 2025B'ezierGS: Dynamic Urban Scene Reconstruction with B'ezier Curve Gaussian Splattingcited by 5
  19. 2026HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Drivingcited by 4
  20. 2026MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systemscited by 4
  21. 2026Scaling Self-Play for End-to-End Drivingcited by 4
  22. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  23. 2025MADrive: Memory-Augmented Driving Scene Modelingcited by 3
  24. 2025D2GS: Dense Depth Regularization for LiDAR-free Urban Scene Reconstructioncited by 3
  25. 2026UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modelingcited by 3
  26. 2026TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrationscited by 3
  27. 2025Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimationcited by 2
  28. 2025SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restorationcited by 2
  29. 2026FreeFix: Boosting 3D Gaussian Splatting via Fine-Tuning-Free Diffusion Modelscited by 2
  30. 2026Pictura: Perspective-View Self-Play at Scale for Drivingcited by 2
  31. 2026WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Drivingcited by 2
  32. 2025SIMSplat: Language-Aligned 4D Gaussian Splatting for Driving Scenario Generationcited by 1
  33. 2025VAD-GS: Visibility-Aware Densification for 3D Gaussian Splatting in Dynamic Urban Scenescited by 1
  34. 2025Hybrid Gaussian Splatting for Novel Urban View Synthesiscited by 1
  35. 2025FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gaincited by 1
  36. 2025HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Drivingcited by 1
  37. 2026Unified Sensor Simulation for Autonomous Drivingcited by 1
  38. 2026M^3: Dense Matching Meets Multi-View Foundation Models for Monocular Gaussian Splatting SLAMcited by 1

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
  2. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  3. 2026VehDyn: A Driving World Model Benchmark for Vehicle Dynamics
  4. 2025ReSim: Reliable World Simulation for Autonomous Driving
  5. 2026Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
  6. 2024XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis
  7. 2026Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes
  8. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving