# Pyramid Diffusion for Fine 3D Large Scene Generation

arXiv: https://arxiv.org/abs/2311.12085 · HTML: https://mlsanity.com/paper/2311.12085/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 43)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [PointAvatar: Deformable Point-based Head Avatars from Videos](https://mlsanity.com/paper/2212.08377/) (2022)
- [Fast Training of Diffusion Models with Masked Transformers](https://mlsanity.com/paper/2306.09305/) (2023)
- [Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review](https://mlsanity.com/paper/2005.09830/) (2020)
- [Learning to Generate Realistic LiDAR Point Clouds](https://mlsanity.com/paper/2209.03954/) (2022)
- [The Power of Points for Modeling Humans in Clothing](https://mlsanity.com/paper/2109.01137/) (2021)

## Built on by (most cited first, 30 of 44)

- [UniScene: Unified Occupancy-centric Driving Scene Generation](https://mlsanity.com/paper/2412.05435/) (2024)
- [OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving](https://mlsanity.com/paper/2405.20337/) (2024)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models](https://mlsanity.com/paper/2412.03934/) (2024)
- [LT3SD: Latent Trees for 3D Scene Diffusion](https://mlsanity.com/paper/2409.08215/) (2024)
- [DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes](https://mlsanity.com/paper/2410.18084/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint](https://mlsanity.com/paper/2410.15391/) (2024)
- [X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability](https://mlsanity.com/paper/2506.13558/) (2025)
- [NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes](https://mlsanity.com/paper/2503.16375/) (2025)
- [SPATIALGEN: Layout-guided 3D Indoor Scene Generation](https://mlsanity.com/paper/2509.14981/) (2025)
- [[MASK] is All You Need](https://mlsanity.com/paper/2412.06787/) (2024)
- [Controllable 3D Outdoor Scene Generation via Scene Graphs](https://mlsanity.com/paper/2503.07152/) (2025)
- [Conditional Panoramic Image Generation via Masked Autoregressive Modeling](https://mlsanity.com/paper/2505.16862/) (2025)
- [Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method](https://mlsanity.com/paper/2510.22973/) (2025)
- [HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis](https://mlsanity.com/paper/2506.19291/) (2025)
- [SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation](https://mlsanity.com/paper/2602.22785/) (2026)
- [Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving](https://mlsanity.com/paper/2503.21449/) (2025)
- [Generating 360deg Video is What You Need For a 3D Scene](https://mlsanity.com/paper/2504.02045/) (2025)
- [Extend3D: Town-Scale 3D Generation](https://mlsanity.com/paper/2603.29387/) (2026)
- [SyncSDE: A Probabilistic Framework for Diffusion Synchronization](https://mlsanity.com/paper/2503.21555/) (2025)
- [Constructing a 3D Scene from a Single Image](https://mlsanity.com/paper/2505.15765/) (2025)
- [CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation](https://mlsanity.com/paper/2510.13245/) (2025)
- [Self-Evolving 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2512.08905/) (2025)
- [SSEditor: Controllable Mask-to-Scene Generation with Diffusion Model](https://mlsanity.com/paper/2411.12290/) (2024)
- [MSF: Efficient Diffusion Model Via Multi-Scale Latent Factorize](https://mlsanity.com/paper/2501.13349/) (2025)
- [Scene Coordinate Reconstruction Priors](https://mlsanity.com/paper/2510.12387/) (2025)
- [DVD: Discrete Voxel Diffusion for 3D Generation and Editing](https://mlsanity.com/paper/2605.07971/) (2026)
- [Map Imagination Like Blind Humans: Group Diffusion Model for Robotic Map Generation](https://mlsanity.com/paper/2412.16908/) (2024)
- [Neural Fidelity Calibration for Informative Sim-to-Real Adaptation](https://mlsanity.com/paper/2504.08604/) (2025)

## Abstract (arXiv)

Diffusion models have shown remarkable results in generating 2D images and small-scale 3D objects. However, their application to the synthesis of large-scale 3D scenes has been rarely explored. This is mainly due to the inherent complexity and bulky size of 3D scenery data, particularly outdoor scenes, and the limited availability of comprehensive real-world datasets, which makes training a stable scene diffusion model challenging. In this work, we explore how to effectively generate large-scale 3D scenes using the coarse-to-fine paradigm. We introduce a framework, the Pyramid Discrete Diffusion model (PDD), which employs scale-varied diffusion models to progressively generate high-quality outdoor scenes. Experimental results of PDD demonstrate our successful exploration in generating 3D scenes both unconditionally and conditionally. We further showcase the data compatibility of the PDD model, due to its multi-scale architecture: a PDD model trained on one dataset can be easily fine-tuned with another dataset. Code is available at https://github.com/yuhengliu02/pyramid-discrete-diffusion.