# PanoDreamer: Consistent Text to 360-Degree Scene Generation

arXiv: https://arxiv.org/abs/2504.05152 · HTML: https://mlsanity.com/paper/2504.05152/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Phasic Policy Gradient](https://mlsanity.com/paper/2009.04416/) (2020)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [Proximal Policy Optimization via Enhanced Exploration Efficiency](https://mlsanity.com/paper/2011.05525/) (2020)
- [Uncertainty-Aware Policy Optimization: A Robust, Adaptive Trust Region Approach](https://mlsanity.com/paper/2012.10791/) (2020)
- [Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning](https://mlsanity.com/paper/1912.05109/) (2019)
- [DNA: Proximal Policy Optimization with a Dual Network Architecture](https://mlsanity.com/paper/2206.10027/) (2022)

## Built on by (most cited first, 3 of 3)

- [ActWorld: From Explorable to Interactive World Model via Action-Aware Memory](https://mlsanity.com/paper/2606.17730/) (2026)
- [EmoWorld: A Decoupled Affective Field for Controllable Emotional Video Generation](https://mlsanity.com/paper/2608.06231/) (2026)
- [TASE: Truncation-Aware Semantic Embeddings for 3D Scene Understanding and Editing](https://mlsanity.com/paper/2606.03314/) (2026)

## Abstract (arXiv)

Automatically generating a complete 3D scene from a text description, a reference image, or both has significant applications in fields like virtual reality and gaming. However, current methods often generate low-quality textures and inconsistent 3D structures. This is especially true when extrapolating significantly beyond the field of view of the reference image. To address these challenges, we propose PanoDreamer, a novel framework for consistent, 3D scene generation with flexible text and image control. Our approach employs a large language model and a warp-refine pipeline, first generating an initial set of images and then compositing them into a 360-degree panorama. This panorama is then lifted into 3D to form an initial point cloud. We then use several approaches to generate additional images, from different viewpoints, that are consistent with the initial point cloud and expand/refine the initial point cloud. Given the resulting set of images, we utilize 3D Gaussian Splatting to create the final 3D scene, which can then be rendered from different viewpoints. Experiments demonstrate the effectiveness of PanoDreamer in generating high-quality, geometrically consistent 3D scenes.