mlsanityData to 14 Oct 2026

FreeVS: Generative View Synthesis on Free Driving Trajectory

arXiv 2410.18079 ↗First posted 23 Oct 2024cs.CVcited by 53

Not on the map. Why?

Abstract (arXiv)

Existing reconstruction-based novel view synthesis methods for driving scenes focus on synthesizing camera views along the recorded trajectory of the ego vehicle. Their image rendering performance will severely degrade on viewpoints falling out of the recorded trajectory, where camera rays are untrained. We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes. To control the generation results to be 3D consistent with the real scenes and accurate in viewpoint pose, we propose the pseudo-image representation of view priors to control the generation process. Viewpoint transformation simulation is applied on pseudo-images to simulate camera movement in each direction. Once trained, FreeVS can be applied to any validation sequences without reconstruction process and synthesis views on novel trajectories. Moreover, we propose two new challenging benchmarks tailored to driving scenes, which are novel camera synthesis and novel trajectory synthesis, emphasizing the freedom of viewpoints. Given that no ground truth images are available on novel trajectories, we also propose to evaluate the consistency of images synthesized on novel trajectories with 3D perception models. Experiments on the Waymo Open Dataset show that FreeVS has a strong image synthesis performance on both the recorded trajectories and novel trajectories. Project Page: https://freevs24.github.io/

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 48 of its 53 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

48 of 53 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2022A ConvNet for the 2020scited by 3,579
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  11. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  12. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
36 more
  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  2. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  3. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  4. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  5. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  6. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  7. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 476
  8. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  9. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 398
  10. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  11. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  12. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 318
  13. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  14. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  15. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  16. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 274
  17. 2021Urban Radiance Fieldscited by 263
  18. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  19. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 221
  20. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 217
  21. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  22. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  23. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 150
  24. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  25. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  26. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 142
  27. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  28. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  29. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  30. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 90
  31. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55
  32. 2021Neural Point Light Fieldscited by 51
  33. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Priorcited by 36
  34. 2022MV-FCOS3D++: Multi-View Camera-Only 4D Object Detection with Pretrained Monocular Backbonescited by 29
  35. 2022LET-3D-AP: Longitudinal Error Tolerant 3D Average Precision for Camera-Only 3D Detectioncited by 27
  36. 2023MapNeRF: Incorporating Map Priors into Neural Radiance Fields for Driving View Simulationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 26; 2026: 21.

53 papers.

Sort:

By area:

  1. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  2. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
  3. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 90
  4. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 40
  5. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  6. 2024FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenescited by 30
  7. 2024Splatter-360: Generalizable 360circ Gaussian Splatting for Wide-baseline Panoramic Imagescited by 24
  8. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  9. 2024Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Modelcited by 18
  10. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  11. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  12. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generationcited by 14
38 more
  1. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 13
  2. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  3. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  4. 2025MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstructioncited by 8
  5. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  6. 2025CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Renderingcited by 5
  7. 2025Spatial Retrieval Augmented Autonomous Drivingcited by 5
  8. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesiscited by 4
  9. 2025FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Renderingcited by 4
  10. 2025ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty-Guided Generative Priorscited by 4
  11. 2026RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Spacecited by 4
  12. 2026GenAssets: Generating in-the-wild 3D Assets in Latent Spacecited by 4
  13. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  14. 2026Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulationcited by 3
  15. 2025RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priorscited by 2
  16. 2025LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priorscited by 2
  17. 2025SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restorationcited by 2
  18. 2026FreeFix: Boosting 3D Gaussian Splatting via Fine-Tuning-Free Diffusion Modelscited by 2
  19. 2025FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gaincited by 1
  20. 2025ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectoriescited by 1
  21. 2026GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generationcited by 1
  22. 2026GS^2: Graph-based Spatial Distribution Optimization for Compact 3D Gaussian Splattingcited by 1
  23. 2026Geo-EVS: Geometry-Conditioned Extrapolative View Synthesis for Autonomous Drivingcited by 1
  24. 2026Diffusion-guided Generalizable Enhancer for Urban Scene Reconstructioncited by 1
  25. 2026Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulationcited by 1
  26. 2025ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Modelscited by 0
  27. 2025Dynamic Novel View Synthesis in High Dynamic Rangecited by 0
  28. 2025ViSE: A Systematic Approach to Vision-Only Street-View Extrapolationcited by 0
  29. 2025FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesiscited by 0
  30. 2025SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesiscited by 0
  31. 2026DriveFix: Spatio-Temporally Coherent Driving Scene Restorationcited by 0
  32. 2026VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigmcited by 0
  33. 2026ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generationcited by 0
  34. 2026ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenescited by 0
  35. 2026Effective Multi-sensor Conditioning for Street-view Novel-view Synthesiscited by 0
  36. 2026DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulationcited by 0
  37. 2026How Can Driving World Models Do Counterfactual Prediction?cited by 0
  38. 2026GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generationcited by 0

50 most cited of 53 All 53 (JSON)