# SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

arXiv: https://arxiv.org/abs/2410.00337 · HTML: https://mlsanity.com/paper/2410.00337/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 39)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2302.04867/) (2023)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023)
- [MagicDrive: Street View Generation with Diverse 3D Geometry Control](https://mlsanity.com/paper/2310.02601/) (2023)
- [Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models](https://mlsanity.com/paper/2303.11989/) (2023)
- [LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2311.13384/) (2023)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [Panacea: Panoramic and Controllable Video Generation for Autonomous Driving](https://mlsanity.com/paper/2311.16813/) (2023)
- [FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation](https://mlsanity.com/paper/2307.01492/) (2023)
- [WonderJourney: Going from Anywhere to Everywhere](https://mlsanity.com/paper/2312.03884/) (2023)
- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout](https://mlsanity.com/paper/2308.01661/) (2023)
- [CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.07724/) (2022)
- [WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation](https://mlsanity.com/paper/2312.02934/) (2023)

## Built on by (most cited first, 14 of 14)

- [UniScene: Unified Occupancy-centric Driving Scene Generation](https://mlsanity.com/paper/2412.05435/) (2024)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability](https://mlsanity.com/paper/2506.13558/) (2025)
- [Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion](https://mlsanity.com/paper/2507.04403/) (2025)
- [ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance](https://mlsanity.com/paper/2504.16464/) (2025)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving](https://mlsanity.com/paper/2512.14225/) (2025)
- [Sat2City v2: Native 3D City Asset Generation from a Single Satellite Image](https://mlsanity.com/paper/2606.24138/) (2026)
- [OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction](https://mlsanity.com/paper/2410.04932/) (2024)
- [MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer](https://mlsanity.com/paper/2508.14327/) (2025)
- [XYZCylinder: Towards Compatible Feed-Forward 3D Gaussian Splatting for Driving Scenes via Unified Cylinder Lifting Method](https://mlsanity.com/paper/2510.07856/) (2025)
- [ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction](https://mlsanity.com/paper/2604.01081/) (2026)
- [MIVIFI: Bridging Perspective and Fisheye Domains for Training Multi-View Fisheye Image Generation Models](https://mlsanity.com/paper/2608.23140/) (2026)

## Abstract (arXiv)

The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric-controlled images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. As a result, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.