# Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction

arXiv: https://arxiv.org/abs/2311.18695 · HTML: https://mlsanity.com/paper/2311.18695/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization](https://mlsanity.com/paper/2303.07418/) (2023)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation](https://mlsanity.com/paper/1901.03861/) (2019)
- [HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features](https://mlsanity.com/paper/2011.11498/) (2020)
- [LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image](https://mlsanity.com/paper/1803.08999/) (2018)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets](https://mlsanity.com/paper/2007.12868/) (2020)
- [DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama](https://mlsanity.com/paper/1811.11977/) (2018)

## Built on by (most cited first, 2 of 2)

- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [OB3D: A New Dataset for Benchmarking Omnidirectional 3D Reconstruction Using Blender](https://mlsanity.com/paper/2505.20126/) (2025)

## Abstract (arXiv)

State-of-the-art single-view 360-degree room layout reconstruction methods formulate the problem as a high-level 1D (per-column) regression task. On the other hand, traditional low-level 2D layout segmentation is simpler to learn and can represent occluded regions, but it requires complex post-processing for the targeting layout polygon and sacrifices accuracy. We present Seg2Reg to render 1D layout depth regression from the 2D segmentation map in a differentiable and occlusion-aware way, marrying the merits of both sides. Specifically, our model predicts floor-plan density for the input equirectangular 360-degree image. Formulating the 2D layout representation as a density field enables us to employ `flattened' volume rendering to form 1D layout depth regression. In addition, we propose a novel 3D warping augmentation on layout to improve generalization. Finally, we re-implement recent room layout reconstruction methods into our codebase for benchmarking and explore modern backbones and training techniques to serve as the strong baseline. Our model significantly outperforms previous arts. The code will be made available upon publication.