# SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis arXiv: https://arxiv.org/abs/2512.22706 ยท HTML: https://mlsanity.com/paper/2512.22706/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. - Builds on 28 of its 32 references that are in the map. - First posted on arXiv 27 Dec 2025. ## Placed in - Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/) ## Builds on (most cited first, 28 of 28) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025) - [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024) - [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024) - [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024) - [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023) - [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023) - [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025) - [Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting](https://mlsanity.com/paper/2401.01339/) (2024) - [Video Depth Anything: Consistent Depth Estimation for Super-Long Videos](https://mlsanity.com/paper/2501.12375/) (2025) - [OmniRe: Omni Urban Scene Reconstruction](https://mlsanity.com/paper/2408.16760/) (2024) - [Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Rendering](https://mlsanity.com/paper/2311.18561/) (2023) - [ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration](https://mlsanity.com/paper/2411.19548/) (2024) - [Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images](https://mlsanity.com/paper/2503.13439/) (2025) - [FreeVS: Generative View Synthesis on Free Driving Trajectory](https://mlsanity.com/paper/2410.18079/) (2024) - [StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models](https://mlsanity.com/paper/2412.13188/) (2024) - [FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes](https://mlsanity.com/paper/2412.03566/) (2024) - [R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation](https://mlsanity.com/paper/2506.07826/) (2025) - [DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes](https://mlsanity.com/paper/2412.19458/) (2024) - [GenMM: Geometrically and Temporally Consistent Multimodal Data Generation for Video and LiDAR](https://mlsanity.com/paper/2406.10722/) (2024) - [Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles](https://mlsanity.com/paper/2412.14494/) (2024) ## Abstract (arXiv) 3D Asset insertion and novel view synthesis (NVS) are key components for autonomous driving simulation, enhancing the diversity of training data. With better training data that is diverse and covers a wide range of situations, including long-tailed driving scenarios, autonomous driving models can become more robust and safer. This motivates a unified simulation framework that can jointly handle realistic integration of inserted 3D assets and NVS. Recent 3D asset reconstruction methods enable reconstruction of dynamic actors from video, supporting their re-insertion into simulated driving scenes. While the overall structure and appearance can be accurate, it still struggles to capture the realism of 3D assets through lighting or shadows, particularly when inserted into scenes. In parallel, recent advances in NVS methods have demonstrated promising results in synthesizing viewpoints beyond the originally recorded trajectories. However, existing approaches largely treat asset insertion and NVS capabilities in isolation. To allow for interaction with the rest of the scene and to enable more diverse creation of new scenarios for training, realistic 3D asset insertion should be combined with NVS. To address this, we present SCPainter (Street Car Painter), a unified framework which integrates 3D Gaussian Splat (GS) car asset representations and 3D scene point clouds with diffusion-based generation to jointly enable realistic 3D asset insertion and NVS. The 3D GS assets and 3D scene point clouds are projected together into novel views, and these projections are used to condition a diffusion model to generate high quality images. Evaluation on the Waymo Open Dataset demonstrate the capability of our framework to enable 3D asset insertion and NVS, facilitating the creation of diverse and realistic driving data.