# DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion

arXiv: https://arxiv.org/abs/2410.24203 · HTML: https://mlsanity.com/paper/2410.24203/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 58 of its 77 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 58)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation](https://mlsanity.com/paper/2403.14621/) (2024)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction](https://mlsanity.com/paper/2406.06521/) (2024)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2312.02201/) (2023)
- [MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion](https://mlsanity.com/paper/2307.01097/) (2023)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers](https://mlsanity.com/paper/2406.10163/) (2024)
- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy](https://mlsanity.com/paper/2403.06467/) (2024)

## Built on by (most cited first, 30 of 38)

- [HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels](https://mlsanity.com/paper/2507.21809/) (2025)
- [PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation](https://mlsanity.com/paper/2411.14423/) (2024)
- [Matrix-3D: Omnidirectional Explorable 3D World Generation](https://mlsanity.com/paper/2508.08086/) (2025)
- [WonderTurbo: Generating Interactive 3D World in 0.72 Seconds](https://mlsanity.com/paper/2504.02261/) (2025)
- [Splatter-360: Generalizable 360circ Gaussian Splatting for Wide-baseline Panoramic Images](https://mlsanity.com/paper/2412.06250/) (2024)
- [PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting](https://mlsanity.com/paper/2412.12096/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training](https://mlsanity.com/paper/2510.11712/) (2025)
- [What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?](https://mlsanity.com/paper/2505.22129/) (2025)
- [DreamCube: 3D Panorama Generation via Multi-plane Synchronization](https://mlsanity.com/paper/2506.17206/) (2025)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [Omni2: Unifying Omnidirectional Image Generation and Editing in an Omni Model](https://mlsanity.com/paper/2504.11379/) (2025)
- [Conditional Panoramic Image Generation via Masked Autoregressive Modeling](https://mlsanity.com/paper/2505.16862/) (2025)
- [CamPVG: Camera-Controlled Panoramic Video Generation with Epipolar-Aware Diffusion](https://mlsanity.com/paper/2509.19979/) (2025)
- [WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration](https://mlsanity.com/paper/2506.20590/) (2025)
- [PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion](https://mlsanity.com/paper/2509.24997/) (2025)
- [360Anything: Geometry-Free Lifting of Images and Videos to 360deg](https://mlsanity.com/paper/2601.16192/) (2026)
- [OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation](https://mlsanity.com/paper/2412.09623/) (2024)
- [Generating 360deg Video is What You Need For a 3D Scene](https://mlsanity.com/paper/2504.02045/) (2025)
- [OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback](https://mlsanity.com/paper/2511.00510/) (2025)
- [GaMO: Geometry-aware Multi-view Diffusion Outpainting for Sparse-View 3D Reconstruction](https://mlsanity.com/paper/2512.25073/) (2025)
- [PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptation](https://mlsanity.com/paper/2509.11092/) (2025)
- [PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion](https://mlsanity.com/paper/2412.04827/) (2024)
- [DreamAnywhere: Object-Centric Panoramic 3D Scene Generation](https://mlsanity.com/paper/2506.20367/) (2025)
- [SpheRoPE: Zero-Shot Optimization-Free 360 Panorama Generation with Spherical RoPE](https://mlsanity.com/paper/2606.32033/) (2026)
- [Pano2World: End-to-End 3D Generation via Unified Multi-View Sequences](https://mlsanity.com/paper/2607.00832/) (2026)
- [CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage](https://mlsanity.com/paper/2605.15597/) (2026)
- [JoLT: Joint Latent Trajectories for Context-Guided High-Resolution Tiled Generation](https://mlsanity.com/paper/2608.15395/) (2026)
- [A Survey on Text-Driven 360-Degree Panorama Generation](https://mlsanity.com/paper/2502.14799/) (2025)
- [Generative Panoramic Image Stitching](https://mlsanity.com/paper/2507.07133/) (2025)

## Abstract (arXiv)

Diffusion-based methods have achieved remarkable achievements in 2D image or 3D object generation, however, the generation of 3D scenes and even 360circ images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-view images. To address these issues, we first establish a large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose a novel text-driven panoramic generation framework, termed DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune a single-view text-to-panorama diffusion model with LoRA on the established panoramic video-text dataset. We further design a spherical epipolar-aware multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images. Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text descriptions and camera poses.