# PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image

arXiv: https://arxiv.org/abs/1812.04072 · HTML: https://mlsanity.com/paper/1812.04072/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2018.

## Built on by (most cited first, 30 of 105)

- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)
- [Map-free Visual Relocalization: Metric Pose Relative to a Single Image](https://mlsanity.com/paper/2210.05494/) (2022)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model](https://mlsanity.com/paper/2403.13064/) (2024)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [iDisc: Internal Discretization for Monocular Depth Estimation](https://mlsanity.com/paper/2304.06334/) (2023)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes](https://mlsanity.com/paper/2206.08423/) (2022)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [NDDepth: Normal-Distance Assisted Monocular Depth Estimation](https://mlsanity.com/paper/2309.10592/) (2023)
- [Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes](https://mlsanity.com/paper/2207.08656/) (2022)
- [Planar Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2103.14644/) (2021)
- [Towards Real-World Category-level Articulation Pose Estimation](https://mlsanity.com/paper/2105.03260/) (2021)
- [StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation](https://mlsanity.com/paper/2108.08574/) (2021)
- [Understanding 3D Object Articulation in Internet Videos](https://mlsanity.com/paper/2203.16531/) (2022)
- [PlaneTR: Structure-Guided Transformers for 3D Plane Recovery](https://mlsanity.com/paper/2107.13108/) (2021)
- [Understanding 3D Object Interaction from a Single Image](https://mlsanity.com/paper/2305.09664/) (2023)
- [A General Protocol to Probe Large Vision Models for 3D Physical Understanding](https://mlsanity.com/paper/2310.06836/) (2023)
- [PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2203.12082/) (2022)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [PlaneFormers: From Sparse View Planes to 3D Reconstruction](https://mlsanity.com/paper/2208.04307/) (2022)
- [Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences](https://mlsanity.com/paper/2404.06337/) (2024)
- [Fully Convolutional Line Parsing](https://mlsanity.com/paper/2104.11207/) (2021)
- [NeurMiPs: Neural Mixture of Planar Experts for View Synthesis](https://mlsanity.com/paper/2204.13696/) (2022)
- [NOPE-SAC: Neural One-Plane RANSAC for Sparse-View Planar 3D Reconstruction](https://mlsanity.com/paper/2211.16799/) (2022)
- [RoRD: Rotation-Robust Descriptors and Orthographic Views for Local Feature Matching](https://mlsanity.com/paper/2103.08573/) (2021)
- [From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction](https://mlsanity.com/paper/2203.08122/) (2022)
- [Unsupervised Homography Estimation with Coplanarity-Aware GAN](https://mlsanity.com/paper/2205.03821/) (2022)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)

## Abstract (arXiv)

This paper proposes a deep neural architecture, PlaneRCNN, that detects and reconstructs piecewise planar surfaces from a single RGB image. PlaneRCNN employs a variant of Mask R-CNN to detect planes with their plane parameters and segmentation masks. PlaneRCNN then jointly refines all the segmentation masks with a novel loss enforcing the consistency with a nearby view during training. The paper also presents a new benchmark with more fine-grained plane segmentations in the ground-truth, in which, PlaneRCNN outperforms existing state-of-the-art methods with significant margins in the plane detection, segmentation, and reconstruction metrics. PlaneRCNN makes an important step towards robust plane extraction, which would have an immediate impact on a wide range of applications including Robotics, Augmented Reality, and Virtual Reality.