# Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles

arXiv: https://arxiv.org/abs/2412.14494 · HTML: https://mlsanity.com/paper/2412.14494/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 17 of 17)

- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model](https://mlsanity.com/paper/2311.09217/) (2023)
- [PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction](https://mlsanity.com/paper/2311.12024/) (2023)
- [StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views](https://mlsanity.com/paper/2306.04988/) (2023)
- [Consistent123: Improve Consistency for One Image to 3D Object Synthesis](https://mlsanity.com/paper/2310.08092/) (2023)
- [IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation](https://mlsanity.com/paper/2402.08682/) (2024)
- [Free3D: Consistent Novel View Synthesis without 3D Representation](https://mlsanity.com/paper/2312.04551/) (2023)
- [3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views](https://mlsanity.com/paper/2406.04875/) (2024)
- [Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models](https://mlsanity.com/paper/2406.09292/) (2024)
- [Real3D: Scaling Up Large Reconstruction Models with Real-World Images](https://mlsanity.com/paper/2406.08479/) (2024)
- [Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images](https://mlsanity.com/paper/2307.14009/) (2023)

## Built on by (most cited first, 10 of 10)

- [Orient Anything V2: Unifying Orientation and Rotation Understanding](https://mlsanity.com/paper/2601.05573/) (2026)
- [MObI: Multimodal Object Inpainting Using Diffusion Models](https://mlsanity.com/paper/2501.03173/) (2025)
- [AutoScape: Geometry-Consistent Long-Horizon Scene Generation](https://mlsanity.com/paper/2510.20726/) (2025)
- [3DCarGen: Scalable 3D Car Generation via 3D-consistent Multi-view Synthesis](https://mlsanity.com/paper/2606.24257/) (2026)
- [Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation](https://mlsanity.com/paper/2507.23058/) (2025)
- [ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Models](https://mlsanity.com/paper/2508.05236/) (2025)
- [SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis](https://mlsanity.com/paper/2512.22706/) (2025)
- [MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving](https://mlsanity.com/paper/2606.24301/) (2026)
- [HiFiVe: High-Fidelity Vehicle Generation Leveraging Auto-Regressive 2D Generative Priors](https://mlsanity.com/paper/2606.25300/) (2026)
- [RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors](https://mlsanity.com/paper/2609.31374/) (2026)

## Abstract (arXiv)

The recent advent of large-scale 3D data, e.g. Objaverse, has led to impressive progress in training pose-conditioned diffusion models for novel view synthesis. However, due to the synthetic nature of such 3D data, their performance drops significantly when applied to real-world images. This paper consolidates a set of good practices to finetune large pretrained models for a real-world task -- harvesting vehicle assets for autonomous driving applications. To this end, we delve into the discrepancies between the synthetic data and real driving data, then develop several strategies to account for them properly. Specifically, we start with a virtual camera rotation of real images to ensure geometric alignment with synthetic data and consistency with the pose manifold defined by pretrained models. We also identify important design choices in object-centric data curation to account for varying object distances in real driving scenes -- learn across varying object scales with fixed camera focal length. Further, we perform occlusion-aware training in latent spaces to account for ubiquitous occlusions in real data, and handle large viewpoint changes by leveraging a symmetric prior. Our insights lead to effective finetuning that results in a 68.8% reduction in FID for novel view synthesis over prior arts.