mlsanityData to 7 Oct 2026

DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models

arXiv 2211.12131 ↗First posted 22 Nov 2022cs.CVcited by 51

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

Scene extrapolation -- the idea of generating novel views by flying into a given image -- is a promising, yet challenging task. For each predicted frame, a joint inpainting and 3D refinement problem has to be solved, which is ill posed and includes a high level of ambiguity. Moreover, training data for long-range scenes is difficult to obtain and usually lacks sufficient views to infer accurate camera poses. We introduce DiffDreamer, an unsupervised framework capable of synthesizing novel views depicting a long camera trajectory while training solely on internet-collected images of nature scenes. Utilizing the stochastic nature of the guided denoising steps, we train the diffusion models to refine projected RGBD images but condition the denoising steps on multiple past and future frames for inference. We demonstrate that image-conditioned diffusion models can effectively perform long-range scene extrapolation while preserving consistency significantly better than prior GAN-based methods. DiffDreamer is a powerful and efficient solution for scene extrapolation, producing impressive results despite limited supervision. Project page: https://primecai.github.io/diffdreamer.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 85 of its 109 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

85 of 109 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  2. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 82

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  12. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
38 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  3. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  7. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  8. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  13. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  14. 2018Demystifying MMD GANscited by 1,268
  15. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  16. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  17. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  18. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  19. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  20. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  21. 2020Neural Sparse Voxel Fieldscited by 959
  22. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  23. 2018Self-Attention Generative Adversarial Networkscited by 809
  24. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  25. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  26. 2020Convolutional Occupancy Networkscited by 760
  27. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  28. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  29. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  30. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  31. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  32. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  33. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  34. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  35. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  36. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  37. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  38. 2019Neural Point-Based Graphicscited by 326

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 19; 2025: 21; 2026: 4.

51 papers.

Sort:

By area:

  1. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  4. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  5. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  6. 2024I-Design: Personalized LLM Interior Designercited by 87
  7. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  8. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  9. 2024Advances in 3D Generation: A Surveycited by 62
  10. 2024MegaScenes: Scene-Level View Synthesis at Scalecited by 58
  11. 2024LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generationcited by 53
  12. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 46
38 more
  1. 2023DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Controlcited by 44
  2. 2025GenFusion: Closing the Loop between Reconstruction and Generation via Videoscited by 41
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  4. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  5. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 34
  6. 2025SynCity: Training-Free Generation of 3D Worldscited by 30
  7. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 30
  8. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Modelscited by 29
  9. 2024Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpaintingcited by 23
  10. 2024Explorative Inbetweening of Time and Spacecited by 21
  11. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  12. 2024Urban Scene Diffusion through Semantic Occupancy Mapcited by 14
  13. 2023Chasing Consistency in Text-to-3D Generation from a Single Imagecited by 13
  14. 2024GenRC: Generative 3D Room Completion from Sparse Image Collectionscited by 9
  15. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  16. 2025ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignmentcited by 7
  17. 2025TRELLISWorld: Training-Free World Generation from Object Generatorscited by 7
  18. 2025GarmentCrafter: Progressive Novel View Synthesis for Single-View 3D Garment Reconstruction and Editingcited by 6
  19. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  20. 2024A Comprehensive Survey on Diffusion Models and Their Applicationscited by 4
  21. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesiscited by 4
  22. 2025Diffusion Distillation With Direct Preference Optimization For Efficient 3D LiDAR Scene Completioncited by 4
  23. 2025MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusioncited by 3
  24. 2025Self-Evolving 3D Scene Generation from a Single Imagecited by 3
  25. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3
  26. 2023NeRF-Enhanced Outpainting for Faithful Field-of-View Extrapolationcited by 2
  27. 2025Constructing a 3D Scene from a Single Imagecited by 2
  28. 2025Remote Sensing-Oriented World Modelcited by 2
  29. 2023HawkI: Homography & Mutual Information Guidance for 3D-free Single Image to Aerial Viewcited by 1
  30. 2023DreamDrone: Text-to-Image Diffusion Models are Zero-shot Perpetual View Generatorscited by 1
  31. 2025Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Imagescited by 1
  32. 2025Learning an Implicit Physics Model for Image-based Fluid Simulationcited by 1
  33. 2025MotionFlow:Learning Implicit Motion Flow for Complex Camera Trajectory Control in Video Generationcited by 1
  34. 2025NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfoldingcited by 1
  35. 2025WonderZoom: Multi-Scale 3D World Generationcited by 1
  36. 2026RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roamingcited by 1
  37. 2024Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Promptcited by 0
  38. 2025Novel View Synthesis using DDIM Inversioncited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models
  2. 2024Diffusing Differentiable Representations
  3. 2023DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing
  4. 2024MultiDiff: Consistent Novel View Synthesis from a Single Image
  5. 2025DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation
  6. 2025DT-NVS: Diffusion Transformers for Novel View Synthesis
  7. 2024Diffusion Models in Low-Level Vision: A Survey
  8. 2024Coherent 3D Scene Diffusion From a Single RGB Image