# PlaneTR: Structure-Guided Transformers for 3D Plane Recovery

arXiv: https://arxiv.org/abs/2107.13108 · HTML: https://mlsanity.com/paper/2107.13108/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image](https://mlsanity.com/paper/1812.04072/) (2018)
- [PolyTransform: Deep Polygon Transformer for Instance Segmentation](https://mlsanity.com/paper/1912.02801/) (2019)
- [PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1804.06278/) (2018)
- [Line Segment Detection Using Transformers without Edges](https://mlsanity.com/paper/2101.01909/) (2021)
- [End-to-End Wireframe Parsing](https://mlsanity.com/paper/1905.03246/) (2019)
- [Holistically-Attracted Wireframe Parsing](https://mlsanity.com/paper/2003.01663/) (2020)
- [SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation](https://mlsanity.com/paper/1905.08598/) (2019)
- [Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding](https://mlsanity.com/paper/1902.09777/) (2019)
- [Learning Attraction Field Representation for Robust Line Segment Detection](https://mlsanity.com/paper/1812.02122/) (2018)
- [EdgeStereo: An Effective Multi-Task Learning Network for Stereo Matching and Edge Detection](https://mlsanity.com/paper/1903.01700/) (2019)
- [Monocular Object and Plane SLAM in Structured Environments](https://mlsanity.com/paper/1809.03415/) (2018)
- [Learning Regional Attraction for Line Segment Detection](https://mlsanity.com/paper/1912.09344/) (2019)
- [Multi-Plane Program Induction with 3D Box Priors](https://mlsanity.com/paper/2011.10007/) (2020)
- [Pixel-Pair Occlusion Relationship Map(P2ORM): Formulation, Inference & Application](https://mlsanity.com/paper/2007.12088/) (2020)
- [Surface Reconstruction from 3D Line Segments](https://mlsanity.com/paper/1911.00451/) (2019)

## Built on by (most cited first, 30 of 33)

- [MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer](https://mlsanity.com/paper/2203.10981/) (2022)
- [Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries](https://mlsanity.com/paper/2211.15658/) (2022)
- [PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2203.12082/) (2022)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [Holistically-Attracted Wireframe Parsing: From Supervised to Self-Supervised Learning](https://mlsanity.com/paper/2210.12971/) (2022)
- [NOPE-SAC: Neural One-Plane RANSAC for Sparse-View Planar 3D Reconstruction](https://mlsanity.com/paper/2211.16799/) (2022)
- [Unsupervised Homography Estimation with Coplanarity-Aware GAN](https://mlsanity.com/paper/2205.03821/) (2022)
- [AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings](https://mlsanity.com/paper/2406.08960/) (2024)
- [PlanarSplatting: Accurate Planar Surface Reconstruction in 3 Minutes](https://mlsanity.com/paper/2412.03451/) (2024)
- [HoW-3D: Holistic 3D Wireframe Perception from a Single Image](https://mlsanity.com/paper/2208.06999/) (2022)
- [Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction](https://mlsanity.com/paper/2303.05937/) (2023)
- [NEAT: Distilling 3D Wireframes from Neural Attraction Fields](https://mlsanity.com/paper/2307.10206/) (2023)
- [RoSI: Recovering 3D Shape Interiors from Few Articulation Images](https://mlsanity.com/paper/2304.06342/) (2023)
- [Polygon Detection for Room Layout Estimation using Heterogeneous Graphs and Wireframes](https://mlsanity.com/paper/2306.12203/) (2023)
- [PlaneRecTR++: Unified Query Learning for Joint 3D Planar Reconstruction and Pose Estimation](https://mlsanity.com/paper/2307.13756/) (2023)
- [AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos](https://mlsanity.com/paper/2411.19950/) (2024)
- [Planar Gaussian Splatting](https://mlsanity.com/paper/2412.01931/) (2024)
- [Towards In-the-wild 3D Plane Reconstruction from a Single Image](https://mlsanity.com/paper/2506.02493/) (2025)
- [PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields](https://mlsanity.com/paper/2401.00871/) (2023)
- [PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-Forward Planar Splatting](https://mlsanity.com/paper/2510.18714/) (2025)
- [MCTS with Refinement for Proposals Selection Games in Scene Understanding](https://mlsanity.com/paper/2207.03204/) (2022)
- [A boundary-aware point clustering approach in Euclidean and embedding spaces for roof plane segmentation](https://mlsanity.com/paper/2309.03722/) (2023)
- [UniPlane: Unified Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2407.03594/) (2024)
- [MonoPlane: Exploiting Monocular Geometric Cues for Generalizable 3D Plane Reconstruction](https://mlsanity.com/paper/2411.01226/) (2024)
- [Neural Mesh Fusion: Unsupervised 3D Planar Surface Understanding](https://mlsanity.com/paper/2402.16739/) (2024)
- [Indoor 3D Reconstruction with an Unknown Camera-Projector Pair](https://mlsanity.com/paper/2407.01945/) (2024)
- [Interacted Planes Reveal 3D Line Mapping](https://mlsanity.com/paper/2602.01296/) (2026)
- [Multi-task Planar Reconstruction with Feature Warping Guidance](https://mlsanity.com/paper/2311.14981/) (2023)
- [PlaneSAM: Multimodal Plane Instance Segmentation Using the Segment Anything Model](https://mlsanity.com/paper/2410.16545/) (2024)
- [Plane Detection and Ranking via Model Information Optimization](https://mlsanity.com/paper/2508.09625/) (2025)

## Abstract (arXiv)

This paper presents a neural network built upon Transformers, namely PlaneTR, to simultaneously detect and reconstruct planes from a single image. Different from previous methods, PlaneTR jointly leverages the context information and the geometric structures in a sequence-to-sequence way to holistically detect plane instances in one forward pass. Specifically, we represent the geometric structures as line segments and conduct the network with three main components: (i) context and line segments encoders, (ii) a structure-guided plane decoder, (iii) a pixel-wise plane embedding decoder. Given an image and its detected line segments, PlaneTR generates the context and line segment sequences via two specially designed encoders and then feeds them into a Transformers-based decoder to directly predict a sequence of plane instances by simultaneously considering the context and global structure cues. Finally, the pixel-wise embeddings are computed to assign each pixel to one predicted plane instance which is nearest to it in embedding space. Comprehensive experiments demonstrate that PlaneTR achieves a state-of-the-art performance on the ScanNet and NYUv2 datasets.