# BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning

arXiv: https://arxiv.org/abs/2112.06179 · HTML: https://mlsanity.com/paper/2112.06179/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 92 references that are in the map.
- First posted on arXiv 12 Dec 2021.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://mlsanity.com/paper/2103.10428/) (2021)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image](https://mlsanity.com/paper/1709.07492/) (2017)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [Learning Depth with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1810.02695/) (2018)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [Learning Guided Convolutional Network for Depth Completion](https://mlsanity.com/paper/1908.01238/) (2019)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion](https://mlsanity.com/paper/1911.05377/) (2019)
- [Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints](https://mlsanity.com/paper/1910.06727/) (2019)
- [Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE](https://mlsanity.com/paper/2103.10022/) (2021)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)

## Built on by (most cited first, 26 of 26)

- [360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model](https://mlsanity.com/paper/2401.06578/) (2024)
- [PanoDiffusion: 360-degree Panorama Outpainting via Diffusion](https://mlsanity.com/paper/2307.03177/) (2023)
- [Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models](https://mlsanity.com/paper/2310.18840/) (2023)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [Imagine360: Immersive 360 Video Generation from Perspective Anchor](https://mlsanity.com/paper/2412.03552/) (2024)
- [PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting](https://mlsanity.com/paper/2412.12096/) (2024)
- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training](https://mlsanity.com/paper/2510.11712/) (2025)
- [PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance](https://mlsanity.com/paper/2408.02157/) (2024)
- [Beyond the Frame: Generating 360 Panoramic Videos from Perspective Videos](https://mlsanity.com/paper/2504.07940/) (2025)
- [What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?](https://mlsanity.com/paper/2505.22129/) (2025)
- [Taming Stable Diffusion for Text to 360deg Panorama Image Generation](https://mlsanity.com/paper/2404.07949/) (2024)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [Omni2: Unifying Omnidirectional Image Generation and Editing in an Omni Model](https://mlsanity.com/paper/2504.11379/) (2025)
- [Conditional Panoramic Image Generation via Masked Autoregressive Modeling](https://mlsanity.com/paper/2505.16862/) (2025)
- [CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Model](https://mlsanity.com/paper/2407.07174/) (2024)
- [OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation](https://mlsanity.com/paper/2412.09623/) (2024)
- [Pantheon360: Taming Digital Twin Generation via 3D-Aware 360deg Video Diffusion](https://mlsanity.com/paper/2605.25449/) (2026)
- [SE360: Semantic Edit in 360^circ Panoramas via Hierarchical Data Construction](https://mlsanity.com/paper/2512.19943/) (2025)
- [SpheRoPE: Zero-Shot Optimization-Free 360 Panorama Generation with Spherical RoPE](https://mlsanity.com/paper/2606.32033/) (2026)
- [A Survey on Text-Driven 360-Degree Panorama Generation](https://mlsanity.com/paper/2502.14799/) (2025)
- [Gimbal360: Canonicalizing Planar Diffusion for Spherical Panorama Completion](https://mlsanity.com/paper/2603.23179/) (2026)
- [Enhancing In-context Panoramic Generation via Geometric-aware Pretraining](https://mlsanity.com/paper/2607.08765/) (2026)
- [Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generation](https://mlsanity.com/paper/2608.20691/) (2026)
- [StreetDiff: Multi-view Street Scenes Generation via Cross-view Consistent Multi-view Stable Diffusion with Structure Prompts](https://mlsanity.com/paper/2609.09890/) (2026)

## Abstract (arXiv)

Providing omnidirectional depth along with RGB information is important for numerous applications, eg, VR/AR. However, as omnidirectional RGB-D data is not always available, synthesizing RGB-D panorama data from limited information of a scene can be useful. Therefore, some prior works tried to synthesize RGB panorama images from perspective RGB images; however, they suffer from limited image quality and can not be directly extended for RGB-D panorama synthesis. In this paper, we study a new problem: RGB-D panorama synthesis under the arbitrary configurations of cameras and depth sensors. Accordingly, we propose a novel bi-modal (RGB-D) panorama synthesis (BIPS) framework. Especially, we focus on indoor environments where the RGB-D panorama can provide a complete 3D model for many applications. We design a generator that fuses the bi-modal information and train it with residual-aided adversarial learning (RDAL). RDAL allows to synthesize realistic indoor layout structures and interiors by jointly inferring RGB panorama, layout depth, and residual depth. In addition, as there is no tailored evaluation metric for RGB-D panorama synthesis, we propose a novel metric to effectively evaluate its perceptual quality. Extensive experiments show that our method synthesizes high-quality indoor RGB-D panoramas and provides realistic 3D indoor models than prior methods. Code will be released upon acceptance.