mlsanityData to 7 Oct 2026

Taming Stable Diffusion for Text to 360deg Panorama Image Generation

arXiv 2404.07949 ↗First posted 11 Apr 2024cs.CVcited by 12

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Generative models, e.g., Stable Diffusion, have enabled the creation of photorealistic images from text prompts. Yet, the generation of 360-degree panorama images from text remains a challenge, particularly due to the dearth of paired text-panorama data and the domain gap between panorama and perspective images. In this paper, we introduce a novel dual-branch diffusion model named PanFusion to generate a 360-degree image from a text prompt. We leverage the stable diffusion model as one branch to provide prior knowledge in natural image generation and register it to another panorama branch for holistic image generation. We propose a unique cross-attention mechanism with projection awareness to minimize distortion during the collaborative denoising process. Our experiments validate that PanFusion surpasses existing methods and, thanks to its dual-branch structure, can integrate additional constraints like room layout for customized panorama outputs. Code is available at https://chengzhag.github.io/publication/panfusion.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 57 of its 66 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

57 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models28 Oct
  2. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions8 Jun
  3. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  4. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture18 May
  5. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models21 Apr
  6. 2023Consistent View Synthesis with Pose-Guided Diffusion Models30 Mar
  7. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2016Improved Techniques for Training GANscited by 2,938
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  13. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  14. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  15. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  16. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  17. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  18. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  19. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  20. 2019HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentationcited by 107
  21. 2023DiffCollage: Parallel Generation of Large Content with Diffusion Modelscited by 100
  22. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  23. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Modelcited by 63
  24. 2021360MonoDepth: High-Resolution 360deg Monocular Depth Estimationcited by 60
  25. 2022StyleLight: HDR Panorama Generation for Lighting Estimation and Editingcited by 59
  26. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 59
  27. 2022Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creationcited by 47
  28. 2023PERF: Panoramic Neural Radiance Field from a Single Panoramacited by 45
  29. 2023360-Degree Panorama Generation from Few Unregistered NFoV Imagescited by 42
  30. 2021ACDNet: Adaptively Combined Dilated Convolution for Monocular Panorama Depth Estimationcited by 39
  31. 2023Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generationcited by 33
  32. 2022Guided Co-Modulated GAN for 360deg Field of View Extrapolationcited by 32
  33. 2021BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learningcited by 26
  34. 2023DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagationcited by 25
  35. 2021Layout-Guided Novel View Synthesis from a Single Indoor Panoramacited by 16
  36. 2022High-Resolution Depth Estimation for 360-degree Panoramas through Perspective and Panoramic Depth Images Registrationcited by 10
  37. 2023Conditional 360-degree Image Synthesis for Immersive Indoor Scene Decorationcited by 9
  38. 2020LayoutMP3D: Layout Annotation of Matterport3Dcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 2.

12 papers.

  1. 2025CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generationcited by 35
  2. 2024Splatter-360: Generalizable 360circ Gaussian Splatting for Wide-baseline Panoramic Imagescited by 23
  3. 2025VideoPanda: Video Panoramic Diffusion with Multi-view Attentioncited by 13
  4. 2024TwinDiffusion: Enhancing Coherence and Efficiency in Panoramic Image Generation with Diffusion Modelscited by 5
  5. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  6. 2024COMOGen: A Controllable Text-to-3D Multi-object Generation Frameworkcited by 2
  7. 2025SE360: Semantic Edit in 360^circ Panoramas via Hierarchical Data Constructioncited by 2
  8. 2026PanoWorld: A Generative Spatial World Model for Consistent Whole-House Panorama Synthesiscited by 2
  9. 2026Pano2World: End-to-End 3D Generation via Unified Multi-View Sequencescited by 2
  10. 2024Diffusing Differentiable Representationscited by 1
  11. 2025SparsyFed: Sparse Adaptive Federated Trainingcited by 1
  12. 2025Drag4D: Align Your Motion with Text-Driven 3D Scene Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?
  2. 2023Diffusion360: Seamless 360 Degree Panoramic Image Generation based on Diffusion Models
  3. 2024Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas
  4. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models
  5. 2025TanDiT: Tangent-Plane Diffusion Transformer for High-Quality 360deg Panorama Generation
  6. 2024DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion
  7. 2025A Survey on Text-Driven 360-Degree Panorama Generation
  8. 2024EdgeFusion: On-Device Text-to-Image Generation