mlsanityData to 7 Oct 2026

UniSim: A Neural Closed-Loop Sensor Simulator

arXiv 2308.01898 ↗First posted 3 Aug 2023cs.CV cs.ROcited by 294

Not on the map. Why?

Cited by 55 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Rigorously testing autonomy systems is essential for making safe self-driving vehicles (SDV) a reality. It requires one to generate safety critical scenarios beyond what can be collected safely in the world, as many scenarios happen rarely on public roads. To accurately evaluate performance, we need to test the SDV on these scenarios in closed-loop, where the SDV and other actors interact with each other at each timestep. Previously recorded driving logs provide a rich resource to build these new scenarios from, but for closed loop evaluation, we need to modify the sensor data based on the new scene configuration and the SDV's decisions, as actors might be added or removed and the trajectories of existing actors and the SDV will differ from the original log. In this paper, we present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle and converts it into a realistic closed-loop multi-sensor simulation. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene, and composites them together to simulate LiDAR and camera data at new viewpoints, with actors added or removed and at new placements. To better handle extrapolated views, we incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions. Our experiments show UniSim can simulate realistic sensor data with small domain gap on downstream tasks. With UniSim, we demonstrate closed-loop evaluation of an autonomy system on safety-critical scenarios as if it were in the real world.

Key facts. Cited by 294 papers in the map (data through 7 Oct 2026). Builds on 72 of its 109 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

72 of 109 references.

References · 72

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  8. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  9. 2016OpenAI Gymcited by 1,684
  10. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  11. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
38 more
  1. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  2. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
  5. 2016HyperNetworkscited by 1,003
  6. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  7. 2020Neural Sparse Voxel Fieldscited by 959
  8. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  9. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  11. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  12. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  13. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  14. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  15. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  16. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  17. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  18. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  19. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  20. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  21. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  22. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  23. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  24. 2019Neural Point-Based Graphicscited by 326
  25. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  26. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  27. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  28. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  29. 2021Urban Radiance Fieldscited by 263
  30. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  31. 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
  32. 2020Free View Synthesiscited by 231
  33. 2018SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloudcited by 228
  34. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 216
  35. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 204
  36. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  37. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  38. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 98; 2025: 101; 2026: 65.

294 papers.

Sort:

By area:

  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  2. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  3. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 218
  4. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  5. 2024GaussianPro: 3D Gaussian Splatting with Progressive Propagationcited by 199
  6. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  7. 2024OmniRe: Omni Urban Scene Reconstructioncited by 159
  8. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  9. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  10. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  11. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 141
  12. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
38 more
  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  4. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
  5. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  6. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  7. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  8. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  9. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  10. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  11. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  12. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  13. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 76
  14. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  15. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  16. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 71
  17. 2024SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Drivingcited by 66
  18. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  19. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  20. 2025Robust Autonomy Emerges from Self-Playcited by 64
  21. 20253D and 4D World Modeling: A Surveycited by 64
  22. 2024NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Drivingcited by 61
  23. 2025A Survey of World Models for Autonomous Drivingcited by 60
  24. 2025Embodied AI Agents: Modeling the Worldcited by 57
  25. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  26. 2024STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenescited by 53
  27. 2024AutoSplat: Constrained Gaussian Splatting for Autonomous Driving Scene Reconstructioncited by 52
  28. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  29. 2024BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Predictioncited by 49
  30. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  31. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  32. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  33. 2023Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perceptioncited by 38
  34. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  35. 2023Real-Time Neural Rasterization for Large Scenescited by 37
  36. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Priorcited by 36
  37. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  38. 2024Dynamic 3D Gaussian Fields for Urban Areascited by 35

50 most cited of 294 All 294 (JSON)