# PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion

arXiv: https://arxiv.org/abs/2412.04827 · HTML: https://mlsanity.com/paper/2412.04827/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 48 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting](https://mlsanity.com/paper/2312.00451/) (2023)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models](https://mlsanity.com/paper/2303.11989/) (2023)
- [LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2311.13384/) (2023)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [WonderJourney: Going from Anywhere to Everywhere](https://mlsanity.com/paper/2312.03884/) (2023)
- [3D Ken Burns Effect from a Single Image](https://mlsanity.com/paper/1909.05483/) (2019)
- [Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields](https://mlsanity.com/paper/2305.11588/) (2023)
- [SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions](https://mlsanity.com/paper/2306.05178/) (2023)
- [RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion](https://mlsanity.com/paper/2404.07199/) (2024)
- [DiffCollage: Parallel Generation of Large Content with Diffusion Models](https://mlsanity.com/paper/2303.17076/) (2023)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians](https://mlsanity.com/paper/2403.19495/) (2024)
- [Learning to Synthesize a 4D RGBD Light Field from a Single Image](https://mlsanity.com/paper/1708.03292/) (2017)
- [PanoDiffusion: 360-degree Panorama Outpainting via Diffusion](https://mlsanity.com/paper/2307.03177/) (2023)
- [SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting](https://mlsanity.com/paper/2109.01068/) (2021)

## Built on by (most cited first, 2 of 2)

- [PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptation](https://mlsanity.com/paper/2509.11092/) (2025)
- [UniSHARP: Universal Sharp Monocular View Synthesis](https://mlsanity.com/paper/2606.07514/) (2026)

## Abstract (arXiv)

In this paper, we present PanoDreamer, a novel method for producing a coherent 360deg 3D scene from a single input image. Unlike existing methods that generate the scene sequentially, we frame the problem as single-image panorama and depth estimation. Once the coherent panoramic image and its corresponding depth are obtained, the scene can be reconstructed by inpainting the small occluded regions and projecting them into 3D space. Our key contribution is formulating single-image panorama and depth estimation as two optimization tasks and introducing alternating minimization strategies to effectively solve their objectives. We demonstrate that our approach outperforms existing techniques in single-image 360deg 3D scene reconstruction in terms of consistency and overall quality.