mlsanityData to 14 Oct 2026

ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration

arXiv 2411.19548 ↗First posted 29 Nov 2024cs.CV cs.AI cs.LGcited by 89

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Closed-loop simulation is crucial for end-to-end autonomous driving. Existing sensor simulation methods (e.g., NeRF and 3DGS) reconstruct driving scenes based on conditions that closely mirror training data distributions. However, these methods struggle with rendering novel trajectories, such as lane changes. Recent works have demonstrated that integrating world model knowledge alleviates these issues. Despite their efficiency, these approaches still encounter difficulties in the accurate representation of more complex maneuvers, with multi-lane shifts being a notable example. Therefore, we introduce ReconDreamer, which enhances driving scene reconstruction through incremental integration of world model knowledge. Specifically, DriveRestorer is proposed to mitigate artifacts via online restoration. This is complemented by a progressive data update strategy designed to ensure high-quality rendering for more complex maneuvers. To the best of our knowledge, ReconDreamer is the first method to effectively render in large maneuvers. Experimental results demonstrate that ReconDreamer outperforms Street Gaussians in the NTA-IoU, NTL-IoU, and FID, with relative improvements by 24.87%, 6.72%, and 29.97%. Furthermore, ReconDreamer surpasses DriveDreamer4D with PVG during large maneuver rendering, as verified by a relative improvement of 195.87% in the NTA-IoU metric and a comprehensive user study.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 70 of its 77 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

70 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  2. 2024OmniRe: Omni Urban Scene Reconstruction29 Aug
  3. 2024Dynamic 3D Gaussian Fields for Urban Areas5 Jun
  4. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving30 May
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  6. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes23 May
  7. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior29 Mar
  8. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  9. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  10. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  11. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 59

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  9. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  12. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
38 more
  1. 2022Planning-oriented Autonomous Drivingcited by 1,087
  2. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  3. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  4. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  5. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 746
  6. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  7. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  8. 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 667
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  10. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  11. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  12. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  13. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  14. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  15. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  16. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  17. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  18. 2021Urban Radiance Fieldscited by 263
  19. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  20. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  21. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  22. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 217
  23. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  24. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 190
  25. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 174
  26. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  27. 2021Efficient Neural Radiance Fields for Interactive Free-viewpoint Videocited by 150
  28. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 150
  29. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  30. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 142
  31. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  32. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  33. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  34. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  35. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  36. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  37. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  38. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 51; 2026: 36.

89 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  2. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  3. 20253D and 4D World Modeling: A Surveycited by 67
  4. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
  5. 2025A Survey of World Models for Autonomous Drivingcited by 60
  6. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  7. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 44
  8. 2026ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrievalcited by 41
  9. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  10. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  11. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 33
  12. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 32
38 more
  1. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  2. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  3. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  4. 2025ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstructioncited by 27
  5. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 26
  6. 2026INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrievalcited by 26
  7. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 24
  8. 2026HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrievalcited by 24
  9. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  10. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  11. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  12. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
  13. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  14. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 15
  15. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  16. 2025HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restorationcited by 13
  17. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 13
  18. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  19. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  20. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 11
  21. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  22. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  23. 2026UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Drivingcited by 9
  24. 2026BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Drivingcited by 9
  25. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  26. 2026Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Contextcited by 8
  27. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  28. 2025RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfercited by 7
  29. 2025WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Explorationcited by 7
  30. 2025Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generationcited by 6
  31. 2026LL-GaussianImage: Efficient Image Representation for Zero-shot Low-Light Enhancement with 2D Gaussian Splattingcited by 6
  32. 2025RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generationcited by 5
  33. 2025GS-RoadPatching: Inpainting Gaussians via 3D Searching and Placing for Driving Scenescited by 5
  34. 20254DSTR: Advancing Generative 4D Gaussians with Spatial-Temporal Rectification for High-Quality and Consistent 4D Generationcited by 5
  35. 2025Neural-Driven Image Editingcited by 4
  36. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 4
  37. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  38. 2025InfoCom: Kilobyte-Scale Communication-Efficient Collaborative Perception with Information Bottleneckcited by 3

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors
  2. 2026ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction
  3. 2026Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
  4. 2025SceneCrafter: Controllable Multi-View Driving Scene Editing
  5. 2026Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
  6. 2025RealEngine: Simulating Autonomous Driving in Realistic Context
  7. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Driving
  8. 2025DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenes