# HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions

arXiv: https://arxiv.org/abs/2407.15187 · HTML: https://mlsanity.com/paper/2407.15187/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 50 of its 57 references that are in the map.
- First posted on arXiv 21 Jul 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)

## Built on by (most cited first, 24 of 24)

- [TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models](https://mlsanity.com/paper/2503.05638/) (2025)
- [LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation](https://mlsanity.com/paper/2408.13252/) (2024)
- [DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion](https://mlsanity.com/paper/2410.24203/) (2024)
- [SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting](https://mlsanity.com/paper/2408.13711/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [VideoPanda: Video Panoramic Diffusion with Multi-view Attention](https://mlsanity.com/paper/2504.11389/) (2025)
- [RoomPainter: View-Integrated Diffusion for Consistent Indoor Scene Texturing](https://mlsanity.com/paper/2412.16778/) (2024)
- [Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting](https://mlsanity.com/paper/2507.18678/) (2025)
- [WorldClaw: Agentic 3D Open-World Generation at Scale](https://mlsanity.com/paper/2608.05248/) (2026)
- [ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignment](https://mlsanity.com/paper/2507.19058/) (2025)
- [ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies](https://mlsanity.com/paper/2506.14315/) (2025)
- [Multi-Scale Diffusion: Enhancing Spatial Layout in High-Resolution Panoramic Image Generation](https://mlsanity.com/paper/2410.18830/) (2024)
- [SonoWorld: From One Image to a 3D Audio-Visual Scene](https://mlsanity.com/paper/2603.28757/) (2026)
- [UniWorld-View: Large-Baseline View Synthesis via Video Diffusion Models](https://mlsanity.com/paper/2608.04701/) (2026)
- [PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion](https://mlsanity.com/paper/2412.04827/) (2024)
- [DreamAnywhere: Object-Centric Panoramic 3D Scene Generation](https://mlsanity.com/paper/2506.20367/) (2025)
- [What if? Emulative Simulation with World Models for Situated Reasoning](https://mlsanity.com/paper/2603.06445/) (2026)
- [SparsyFed: Sparse Adaptive Federated Training](https://mlsanity.com/paper/2504.05153/) (2025)
- [From Fields to Splats: A Cross-Domain Survey of Real-Time Neural Scene Representations](https://mlsanity.com/paper/2509.23555/) (2025)
- [A Survey on Text-Driven 360-Degree Panorama Generation](https://mlsanity.com/paper/2502.14799/) (2025)
- [Drag4D: Align Your Motion with Text-Driven 3D Scene Generation](https://mlsanity.com/paper/2509.21888/) (2025)
- [Stepper: Stepwise Immersive Scene Generation with Multiview Panoramas](https://mlsanity.com/paper/2603.28980/) (2026)
- [CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-Centric 3D Scene Generation](https://mlsanity.com/paper/2607.03819/) (2026)

## Abstract (arXiv)

3D scene generation is in high demand across various domains, including virtual reality, gaming, and the film industry. Owing to the powerful generative capabilities of text-to-image diffusion models that provide reliable priors, the creation of 3D scenes using only text prompts has become viable, thereby significantly advancing researches in text-driven 3D scene generation. In order to obtain multiple-view supervision from 2D diffusion models, prevailing methods typically employ the diffusion model to generate an initial local image, followed by iteratively outpainting the local image using diffusion models to gradually generate scenes. Nevertheless, these outpainting-based approaches prone to produce global inconsistent scene generation results without high degree of completeness, restricting their broader applications. To tackle these problems, we introduce HoloDreamer, a framework that first generates high-definition panorama as a holistic initialization of the full 3D scene, then leverage 3D Gaussian Splatting (3D-GS) to quickly reconstruct the 3D scene, thereby facilitating the creation of view-consistent and fully enclosed 3D scenes. Specifically, we propose Stylized Equirectangular Panorama Generation, a pipeline that combines multiple diffusion models to enable stylized and detailed equirectangular panorama generation from complex text prompts. Subsequently, Enhanced Two-Stage Panorama Reconstruction is introduced, conducting a two-stage optimization of 3D-GS to inpaint the missing region and enhance the integrity of the scene. Comprehensive experiments demonstrated that our method outperforms prior works in terms of overall visual consistency and harmony as well as reconstruction quality and rendering robustness when generating fully enclosed scenes.