# HEAT: Holistic Edge Attention Transformer for Structured Reconstruction

arXiv: https://arxiv.org/abs/2111.15143 · HTML: https://mlsanity.com/paper/2111.15143/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [SpaceNet: A Remote Sensing Dataset and Challenge Series](https://mlsanity.com/paper/1807.01232/) (2018)
- [LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image](https://mlsanity.com/paper/1803.08999/) (2018)
- [Line Segment Detection Using Transformers without Edges](https://mlsanity.com/paper/2101.01909/) (2021)
- [End-to-End Wireframe Parsing](https://mlsanity.com/paper/1905.03246/) (2019)
- [Learning to Parse Wireframes in Images of Man-Made Environments](https://mlsanity.com/paper/2007.07527/) (2020)
- [Holistically-Attracted Wireframe Parsing](https://mlsanity.com/paper/2003.01663/) (2020)
- [PPGNet: Learning Point-Pair Graph for Line Segment Detection](https://mlsanity.com/paper/1905.03415/) (2019)
- [Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path](https://mlsanity.com/paper/1908.06702/) (2019)
- [Learning to Reconstruct 3D Manhattan Wireframes from a Single Image](https://mlsanity.com/paper/1905.07482/) (2019)
- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [Conv-MPN: Convolutional Message Passing Neural Network for Structured Outdoor Architecture Reconstruction](https://mlsanity.com/paper/1912.01756/) (2019)
- [Vectorizing World Buildings: Planar Graph Reconstruction by Primitive Detection and Relationship Inference](https://mlsanity.com/paper/1912.05135/) (2019)
- [Structured Outdoor Architecture Reconstruction by Exploration and Classification](https://mlsanity.com/paper/2108.07990/) (2021)

## Built on by (most cited first, 29 of 29)

- [Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries](https://mlsanity.com/paper/2211.15658/) (2022)
- [PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models](https://mlsanity.com/paper/2306.01461/) (2023)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [PolyRoom: Room-aware Transformer for Floorplan Reconstruction](https://mlsanity.com/paper/2407.10439/) (2024)
- [Beyond the Frontier: Predicting Unseen Walls from Occupancy Grids by Learning from Floor Plans](https://mlsanity.com/paper/2406.09160/) (2024)
- [FRI-Net: Floorplan Reconstruction via Room-wise Implicit Representation](https://mlsanity.com/paper/2407.10687/) (2024)
- [RoIPoly: Vectorized Building Outline Extraction Using Vertex and Logit Embeddings](https://mlsanity.com/paper/2407.14920/) (2024)
- [QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding](https://mlsanity.com/paper/2404.06442/) (2024)
- [ScaleLSD: Scalable Deep Line Segment Detection Streamlined](https://mlsanity.com/paper/2506.09369/) (2025)
- [MCTS with Refinement for Proposals Selection Games in Scene Understanding](https://mlsanity.com/paper/2207.03204/) (2022)
- [Point2Graph: An End-to-end Point Cloud-based 3D Open-Vocabulary Scene Graph for Robot Navigation](https://mlsanity.com/paper/2409.10350/) (2024)
- [UNav: An Infrastructure-Independent Vision-Based Navigation System for People with Blindness and Low vision](https://mlsanity.com/paper/2209.11336/) (2022)
- [Explaining Human Preferences via Metrics for Structured 3D Reconstruction](https://mlsanity.com/paper/2503.08208/) (2025)
- [CAGE: Continuity-Aware edGE Network Unlocks Robust Floorplan Reconstruction](https://mlsanity.com/paper/2509.15459/) (2025)
- [Representation Learning for Sequential Volumetric Design Tasks](https://mlsanity.com/paper/2309.02583/) (2023)
- [BADGR: Bundle Adjustment Diffusion Conditioned by GRadients for Wide-Baseline Floor Plan Reconstruction](https://mlsanity.com/paper/2503.19340/) (2025)
- [Optimized Vectorizing of Building Structures with Switch: High-Efficiency Convolutional Channel-Switch Hybridization Strategy](https://mlsanity.com/paper/2306.15035/) (2023)
- [GeoFormer: A Multi-Polygon Segmentation Transformer](https://mlsanity.com/paper/2411.16616/) (2024)
- [PolyRoof: Precision Roof Polygonization in Urban Residential Building with Graph Neural Networks](https://mlsanity.com/paper/2503.10913/) (2025)
- [FloorSAM: SAM-Guided Floorplan Reconstruction with Semantic-Geometric Fusion](https://mlsanity.com/paper/2509.15750/) (2025)
- [Fast SceneScript: Fast and Accurate Language-Based 3D Scene Understanding via Multi-Token Prediction](https://mlsanity.com/paper/2512.05597/) (2025)
- [Raster2Seq: Polygon Sequence Generation for Floorplan Reconstruction](https://mlsanity.com/paper/2602.09016/) (2026)
- [CornerFormer: Boosting Corner Representation for Fine-Grained Structured Reconstruction](https://mlsanity.com/paper/2304.07072/) (2023)
- [Holistic Geometric Feature Learning for Structured Reconstruction](https://mlsanity.com/paper/2309.09622/) (2023)
- [Unsupervised Roofline Extraction from True Orthophotos for LoD2 Building Model Reconstruction](https://mlsanity.com/paper/2310.01067/) (2023)
- [A-Scan2BIM: Assistive Scan to Building Information Modeling](https://mlsanity.com/paper/2311.18166/) (2023)
- [When Should a Network Emit Geometry, and When Should It Detect It? Readout, Reconciliation, and Representation in Floorplan Vectorization](https://mlsanity.com/paper/2608.25608/) (2026)
- [Diffusion Transformers for Roof Graph Synthesis and Reconstruction](https://mlsanity.com/paper/2608.25652/) (2026)
- [Multimodal Floorplan Encoding: Learning Dense Modality-Invariant Representations](https://mlsanity.com/paper/2609.12723/) (2026)

## Abstract (arXiv)

This paper presents a novel attention-based neural network for structured reconstruction, which takes a 2D raster image as an input and reconstructs a planar graph depicting an underlying geometric structure. The approach detects corners and classifies edge candidates between corners in an end-to-end manner. Our contribution is a holistic edge classification architecture, which 1) initializes the feature of an edge candidate by a trigonometric positional encoding of its end-points; 2) fuses image feature to each edge candidate by deformable attention; 3) employs two weight-sharing Transformer decoders to learn holistic structural patterns over the graph edge candidates; and 4) is trained with a masked learning strategy. The corner detector is a variant of the edge classification architecture, adapted to operate on pixels as corner candidates. We conduct experiments on two structured reconstruction tasks: outdoor building architecture and indoor floorplan planar graph reconstruction. Extensive qualitative and quantitative evaluations demonstrate the superiority of our approach over the state of the art. Code and pre-trained models are available at https://heat-structured-reconstruction.github.io.