# Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View

arXiv: https://arxiv.org/abs/1712.04569 · HTML: https://mlsanity.com/paper/1712.04569/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2017.

## Built on by (most cited first, 25 of 25)

- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Waymo Open Dataset: Panoramic Video Panoptic Segmentation](https://mlsanity.com/paper/2206.07704/) (2022)
- [Learning to See before Learning to Act: Visual Pre-training for Manipulation](https://mlsanity.com/paper/2107.00646/) (2021)
- [Capturing Omni-Range Context for Omnidirectional Segmentation](https://mlsanity.com/paper/2103.05687/) (2021)
- [360+x: A Panoptic Multi-modal Scene Understanding Dataset](https://mlsanity.com/paper/2404.00989/) (2024)
- [Guided Co-Modulated GAN for 360deg Field of View Extrapolation](https://mlsanity.com/paper/2204.07286/) (2022)
- [Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images](https://mlsanity.com/paper/2108.09022/) (2021)
- [LiDAR-based 4D Occupancy Completion and Forecasting](https://mlsanity.com/paper/2310.11239/) (2023)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Navigating to Objects in Unseen Environments by Distance Prediction](https://mlsanity.com/paper/2202.03735/) (2022)
- [Spatially-Varying Outdoor Lighting Estimation from Intrinsics](https://mlsanity.com/paper/2104.04160/) (2021)
- [Environment Predictive Coding for Embodied Agents](https://mlsanity.com/paper/2102.02337/) (2021)
- [3D Scene Geometry Estimation from 360^circ Imagery: A Survey](https://mlsanity.com/paper/2401.09252/) (2024)
- [Indoor Panorama Planar 3D Reconstruction via Divide and Conquer](https://mlsanity.com/paper/2106.14166/) (2021)
- [Glimpse-Attend-and-Explore: Self-Attention for Active Visual Exploration](https://mlsanity.com/paper/2108.11717/) (2021)
- [Enhancement of Novel View Synthesis Using Omnidirectional Image Completion](https://mlsanity.com/paper/2203.09957/) (2022)
- [Single-Shot Cuboids: Geodesics-based End-to-end Manhattan Aligned Layout Estimation from Spherical Panoramas](https://mlsanity.com/paper/2102.03939/) (2021)
- [Beyond the Field-of-View: Enhancing Scene Visibility and Perception with Clip-Recurrent Transformer](https://mlsanity.com/paper/2211.11293/) (2022)
- [CIRCLE: Convolutional Implicit Reconstruction and Completion for Large-scale Indoor Scene](https://mlsanity.com/paper/2111.12905/) (2021)
- [Weakly But Deeply Supervised Occlusion-Reasoned Parametric Road Layouts](https://mlsanity.com/paper/2104.06730/) (2021)
- [Context-aware Padding for Semantic Segmentation](https://mlsanity.com/paper/2109.07854/) (2021)
- [Projective Urban Texturing](https://mlsanity.com/paper/2201.10938/) (2022)
- [HORIZON: High-Resolution Semantically Controlled Panorama Synthesis](https://mlsanity.com/paper/2210.04522/) (2022)
- [MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text](https://mlsanity.com/paper/2404.00345/) (2024)

## Abstract (arXiv)

We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image. To make this possible, Im2Pano3D leverages strong contextual priors learned from large-scale synthetic and real-world indoor scenes. To ease the prediction of 3D structure, we propose to parameterize 3D surfaces with their plane equations and train the model to predict these parameters directly. To provide meaningful training supervision, we use multiple loss functions that consider both pixel level accuracy and global context consistency. Experiments demon- strate that Im2Pano3D is able to predict the semantics and 3D structure of the unobserved scene with more than 56% pixel accuracy and less than 0.52m average distance error, which is significantly better than alternative approaches.