mlsanityData to 14 Oct 2026

FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes

arXiv 2412.03566 ↗First posted 4 Dec 2024cs.CVcited by 30

On the map: World models → World models for autonomous driving

Abstract (arXiv)

We propose FreeSim, a camera simulation method for autonomous driving. FreeSim emphasizes high-quality rendering from viewpoints beyond the recorded ego trajectories. In such viewpoints, previous methods have unacceptable degradation because the training data of these viewpoints is unavailable. To address such data scarcity, we first propose a generative enhancement model with a matched data construction strategy. The resulting model can generate high-quality images in a viewpoint slightly deviated from the recorded trajectories, conditioned on the degraded rendering of this viewpoint. We then propose a progressive reconstruction strategy, which progressively adds generated images of unrecorded views into the reconstruction process, starting from slightly off-trajectory viewpoints and moving progressively farther away. With this progressive generation-reconstruction pipeline, FreeSim supports high-quality off-trajectory view synthesis under large deviations of more than 3 meters.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

38 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  2. 2024OmniRe: Omni Urban Scene Reconstruction29 Aug
  3. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving30 May
  4. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior29 Mar
  5. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  11. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  12. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
20 more
  1. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  2. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
  3. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  4. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  5. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  6. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  7. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  8. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  9. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 220
  10. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 217
  11. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  12. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  13. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 150
  14. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  15. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 143
  16. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 142
  17. 20243DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priorscited by 106
  18. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 53
  19. 2024AutoSplat: Constrained Gaussian Splatting for Autonomous Driving Scene Reconstructioncited by 52
  20. 2023Real-Time Neural Rasterization for Large Scenescited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 21; 2026: 8.

30 papers.

Sort:

By area:

  1. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  2. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
  3. 20253D and 4D World Modeling: A Surveycited by 67
  4. 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videoscited by 46
  5. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  6. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 19
  7. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  8. 2025UniSplat: Unified Spatio-Temporal Fusion via 3D Latent Scaffolds for Dynamic Driving Scene Reconstructioncited by 10
  9. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  10. 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
  11. 2025CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Renderingcited by 5
  12. 2025GS-RoadPatching: Inpainting Gaussians via 3D Searching and Placing for Driving Scenescited by 5
18 more
  1. 2025FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Renderingcited by 4
  2. 2025ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty-Guided Generative Priorscited by 4
  3. 2025Industrial-Grade Sensor Simulation via Gaussian Splatting: A Modular Framework for Scalable Editing and Full-Stack Validationcited by 3
  4. 2025RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priorscited by 2
  5. 2025LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priorscited by 2
  6. 2025SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restorationcited by 2
  7. 2025LidarPainter: One-Step Away From Any Lidar View To Novel Guidancecited by 1
  8. 2025SIMSplat: Language-Aligned 4D Gaussian Splatting for Driving Scenario Generationcited by 1
  9. 2025ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectoriescited by 1
  10. 2026GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generationcited by 1
  11. 2026Diffusion-guided Generalizable Enhancer for Urban Scene Reconstructioncited by 1
  12. 2025FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesiscited by 0
  13. 2025SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesiscited by 0
  14. 2026S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputscited by 0
  15. 2026ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generationcited by 0
  16. 2026ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenescited by 0
  17. 2026FRUC: Feedforward Dynamic Scene Reconstruction from Uncalibrated Collaborative Driving Viewscited by 0
  18. 2026SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Renderingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Driving View Synthesis on Free-form Trajectories with Generative Prior
  2. 2026HelloWorld: Towards Practical Applications of Generative Driving World Models
  3. 2026Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
  4. 2025DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenes
  5. 2025VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
  6. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  7. 2026Digital Twin-Driven Real2Sim2Real: Simulator-Conditioned Generation via Paired Driving-Scene Reconstruction
  8. 20263DCarGen: Scalable 3D Car Generation via 3D-consistent Multi-view Synthesis