# Building extraction with vision transformer

arXiv: https://arxiv.org/abs/2111.15637 · HTML: https://mlsanity.com/paper/2111.15637/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 101 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Remote Sensing Image Change Detection with Transformers](https://mlsanity.com/paper/2103.00208/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery](https://mlsanity.com/paper/2109.08937/) (2021)
- [Visformer: The Vision-friendly Transformer](https://mlsanity.com/paper/2104.12533/) (2021)
- [Deep learning in remote sensing: a review](https://mlsanity.com/paper/1710.03959/) (2017)
- [Rethinking BiSeNet For Real-time Semantic Segmentation](https://mlsanity.com/paper/2104.13188/) (2021)
- [Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images](https://mlsanity.com/paper/2009.02130/) (2020)
- [A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images](https://mlsanity.com/paper/2104.12137/) (2021)
- [MAP-Net: Multi Attending Path Neural Network for Building Footprint Extraction from Remote Sensed Imagery](https://mlsanity.com/paper/1910.12060/) (2019)

## Built on by (most cited first, 22 of 22)

- [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022)
- [CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Model](https://mlsanity.com/paper/2402.03631/) (2024)
- [FlexiMo: A Flexible Remote Sensing Foundation Model](https://mlsanity.com/paper/2503.23844/) (2025)
- [RSBuilding: Towards General Remote Sensing Image Building Extraction and Change Detection with Foundation Model](https://mlsanity.com/paper/2403.07564/) (2024)
- [Building Extraction from Remote Sensing Images via an Uncertainty-Aware Network](https://mlsanity.com/paper/2307.12309/) (2023)
- [LSwinSR: UAV Imagery Super-Resolution based on Linear Swin Transformer](https://mlsanity.com/paper/2303.10232/) (2023)
- [Delineate Anything: Resolution-Agnostic Field Boundary Delineation on Satellite Imagery](https://mlsanity.com/paper/2504.02534/) (2025)
- [Globally Scalable Glacier Mapping by Deep Learning Matches Expert Delineation Accuracy](https://mlsanity.com/paper/2401.15113/) (2024)
- [PolyBuilding: Polygon Transformer for End-to-End Building Extraction](https://mlsanity.com/paper/2211.01589/) (2022)
- [LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Images](https://mlsanity.com/paper/2212.08490/) (2022)
- [Building-road Collaborative Extraction from Remotely Sensed Images via Cross-Interaction](https://mlsanity.com/paper/2307.12256/) (2023)
- [Identifying every building's function in large-scale urban areas with multi-modality remote-sensing data](https://mlsanity.com/paper/2405.05133/) (2024)
- [Deep Learning for Slum Mapping in Remote Sensing Images: A Meta-analysis and Review](https://mlsanity.com/paper/2406.08031/) (2024)
- [UAGLNet: Uncertainty-Aggregated Global-Local Fusion Network with Cooperative CNN-Transformer for Building Extraction](https://mlsanity.com/paper/2512.12941/) (2025)
- [BuildMamba: A Visual State-Space Based Model for Multi-Task Building Segmentation and Height Estimation from Satellite Images](https://mlsanity.com/paper/2603.08523/) (2026)
- [Building and Road Segmentation Using EffUNet and Transfer Learning Approach](https://mlsanity.com/paper/2307.03980/) (2023)
- [Context-Enhanced Detector For Building Detection From Remote Sensing Images](https://mlsanity.com/paper/2310.07638/) (2023)
- [Segmentation of arbitrary features in very high resolution remote sensing imagery](https://mlsanity.com/paper/2412.16046/) (2024)
- [MSSFC-Net:Enhancing Building Interpretation with Multi-Scale Spatial-Spectral Feature Collaboration](https://mlsanity.com/paper/2504.00759/) (2025)
- [Building Extraction from Remote Sensing Imagery under Hazy and Low-light Conditions: Benchmark and Baseline](https://mlsanity.com/paper/2604.15088/) (2026)
- [Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery](https://mlsanity.com/paper/2605.26381/) (2026)
- [UniBuild: Unified Building Mapping From Multi-Source Optical Remote Sensing Imagery With Detail Decoding and Geometry Regularization](https://mlsanity.com/paper/2609.37031/) (2026)

## Abstract (arXiv)

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN) based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modelling global dependencies. Thus, this paper applies the Vision Transformer for building extraction. However, the actual utilization of the Vision Transformer often comes with two limitations. First, the Vision Transformer requires more GPU memory and computational costs compared to CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the Vision Transformer, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel Vision Transformer (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we develop a window-based linear multi-head self-attention to make the complexity of the multi-head self-attention linear with the window size, which strengthens the global context extraction by using large windows and greatly improves the potential of the Vision Transformer in processing large-sized remote sensing images. The proposed method yields state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available.