# DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models

arXiv: https://arxiv.org/abs/2211.12131 · HTML: https://mlsanity.com/paper/2211.12131/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 85 of its 109 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)
- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 85)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)

## Built on by (most cited first, 30 of 51)

- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [WonderJourney: Going from Anywhere to Everywhere](https://mlsanity.com/paper/2312.03884/) (2023)
- [Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control](https://mlsanity.com/paper/2405.17414/) (2024)
- [CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models](https://mlsanity.com/paper/2503.10592/) (2025)
- [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer](https://mlsanity.com/paper/2405.15364/) (2024)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [MegaScenes: Scene-Level View Synthesis at Scale](https://mlsanity.com/paper/2406.11819/) (2024)
- [LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation](https://mlsanity.com/paper/2408.13252/) (2024)
- [Diffusion Self-Distillation for Zero-Shot Customized Image Generation](https://mlsanity.com/paper/2411.18616/) (2024)
- [DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control](https://mlsanity.com/paper/2312.03048/) (2023)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [GenFusion: Closing the Loop between Reconstruction and Generation via Videos](https://mlsanity.com/paper/2503.21219/) (2025)
- [Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion](https://mlsanity.com/paper/2407.13759/) (2024)
- [AnimateAnything: Consistent and Controllable Animation for Video Generation](https://mlsanity.com/paper/2411.10836/) (2024)
- [FlashWorld: High-quality 3D Scene Generation within Seconds](https://mlsanity.com/paper/2510.13678/) (2025)
- [SynCity: Training-Free Generation of 3D Worlds](https://mlsanity.com/paper/2503.16420/) (2025)
- [Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models](https://mlsanity.com/paper/2312.01409/) (2023)
- [Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting](https://mlsanity.com/paper/2404.19758/) (2024)
- [Explorative Inbetweening of Time and Space](https://mlsanity.com/paper/2403.14611/) (2024)
- [WorldGen: From Text to Traversable and Interactive 3D Worlds](https://mlsanity.com/paper/2511.16825/) (2025)
- [Urban Scene Diffusion through Semantic Occupancy Map](https://mlsanity.com/paper/2403.11697/) (2024)
- [Chasing Consistency in Text-to-3D Generation from a Single Image](https://mlsanity.com/paper/2309.03599/) (2023)
- [GenRC: Generative 3D Room Completion from Sparse Image Collections](https://mlsanity.com/paper/2407.12939/) (2024)
- [TRELLISWorld: Training-Free World Generation from Object Generators](https://mlsanity.com/paper/2510.23880/) (2025)
- [WorldClaw: Agentic 3D Open-World Generation at Scale](https://mlsanity.com/paper/2608.05248/) (2026)
- [KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos](https://mlsanity.com/paper/2411.13549/) (2024)
- [ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignment](https://mlsanity.com/paper/2507.19058/) (2025)

## Abstract (arXiv)

Scene extrapolation -- the idea of generating novel views by flying into a given image -- is a promising, yet challenging task. For each predicted frame, a joint inpainting and 3D refinement problem has to be solved, which is ill posed and includes a high level of ambiguity. Moreover, training data for long-range scenes is difficult to obtain and usually lacks sufficient views to infer accurate camera poses. We introduce DiffDreamer, an unsupervised framework capable of synthesizing novel views depicting a long camera trajectory while training solely on internet-collected images of nature scenes. Utilizing the stochastic nature of the guided denoising steps, we train the diffusion models to refine projected RGBD images but condition the denoising steps on multiple past and future frames for inference. We demonstrate that image-conditioned diffusion models can effectively perform long-range scene extrapolation while preserving consistency significantly better than prior GAN-based methods. DiffDreamer is a powerful and efficient solution for scene extrapolation, producing impressive results despite limited supervision. Project page: https://primecai.github.io/diffdreamer.