# Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting arXiv: https://arxiv.org/abs/2401.10564 ยท HTML: https://mlsanity.com/paper/2401.10564/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 71 references that are in the map. - First posted on arXiv 19 Jan 2024. ## Builds on (most cited first, 30 of 35) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020) - [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014) - [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016) - [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018) - [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022) - [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016) - [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020) - [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020) - [High-Fidelity GAN Inversion for Image Attribute Editing](https://mlsanity.com/paper/2109.06590/) (2021) - [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019) - [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021) - [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016) - [Diverse Image Inpainting with Bidirectional and Autoregressive Transformers](https://mlsanity.com/paper/2104.12335/) (2021) - [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019) - [Transformers as Meta-Learners for Implicit Neural Representations](https://mlsanity.com/paper/2208.02801/) (2022) - [Fourier Space Losses for Efficient Perceptual Image Super-Resolution](https://mlsanity.com/paper/2106.00783/) (2021) - [In&Out : Diverse Image Outpainting via GAN Inversion](https://mlsanity.com/paper/2104.00675/) (2021) - [Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation](https://mlsanity.com/paper/2203.14668/) (2022) - [Frequency Domain Image Translation: More Photo-realistic, Better Identity-preserving](https://mlsanity.com/paper/2011.13611/) (2020) - [360-Degree Panorama Generation from Few Unregistered NFoV Images](https://mlsanity.com/paper/2308.14686/) (2023) - [HRDFuse: Monocular 360degDepth Estimation by Collaboratively Learning Holistic-with-Regional Depth Distributions](https://mlsanity.com/paper/2303.11616/) (2023) - [Spectral Regularization for Combating Mode Collapse in GANs](https://mlsanity.com/paper/1908.10999/) (2019) - [Spectral Distribution Aware Image Generation](https://mlsanity.com/paper/2012.03110/) (2020) - [ScanGAN360: A Generative Model of Realistic Scanpaths for 360circ Images](https://mlsanity.com/paper/2103.13922/) (2021) ## Built on by (most cited first, 9 of 9) - [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025) - [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025) - [DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training](https://mlsanity.com/paper/2510.11712/) (2025) - [OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation](https://mlsanity.com/paper/2412.09623/) (2024) - [RealOSR: Latent Guidance Boosts Diffusion-based Real-world Omnidirectional Image Super-Resolutions](https://mlsanity.com/paper/2412.09646/) (2024) - [CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance](https://mlsanity.com/paper/2606.27584/) (2026) - [Unified Panoramic-Gaussian Representation for Monocular 4D Scene Synthesis](https://mlsanity.com/paper/2607.01663/) (2026) - [Enhancing In-context Panoramic Generation via Geometric-aware Pretraining](https://mlsanity.com/paper/2607.08765/) (2026) - [SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera](https://mlsanity.com/paper/2609.33462/) (2026) ## Abstract (arXiv) 360 images, with a field-of-view (FoV) of 180x360, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: `How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360 image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360 images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360 images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.