# Prompt-Driven Building Footprint Extraction in Aerial Images with Offset-Building Model

arXiv: https://arxiv.org/abs/2310.16717 · HTML: https://mlsanity.com/paper/2310.16717/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 81 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 30 of 45)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model](https://mlsanity.com/paper/2306.16269/) (2023)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [Towards Large-Scale Small Object Detection: Survey and Benchmarks](https://mlsanity.com/paper/2207.14096/) (2022)
- [The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot](https://mlsanity.com/paper/2306.16623/) (2023)

## Built on by (most cited first, 3 of 3)

- [PolyFootNet: Extracting Polygonal Building Footprints in Off-Nadir Remote Sensing Images](https://mlsanity.com/paper/2408.08645/) (2024)
- [SiZeUp: Fast 3D Proxy from Aerial Images via Depth Ordinal Loss](https://mlsanity.com/paper/2608.22821/) (2026)
- [LoDEOT: Low-Dimensional and Efficient Offset Tokens for Building Footprint Extraction from Off-Nadir Imagery](https://mlsanity.com/paper/2610.05899/) (2026)

## Abstract (arXiv)

More accurate extraction of invisible building footprints from very-high-resolution (VHR) aerial images relies on roof segmentation and roof-to-footprint offset extraction. Existing methods based on instance segmentation suffer from poor generalization when extended to large-scale data production and fail to achieve low-cost human interaction. This prompt paradigm inspires us to design a promptable framework for roof and offset extraction, and transforms end-to-end algorithms into promptable methods. Within this framework, we propose a novel Offset-Building Model (OBM). Based on prompt prediction, we first discover a common pattern of predicting offsets and tailored Distance-NMS (DNMS) algorithms for offset optimization. To rigorously evaluate the algorithm's capabilities, we introduce a prompt-based evaluation method, where our model reduces offset errors by 16.6% and improves roof Intersection over Union (IoU) by 10.8% compared to other models. Leveraging the common patterns in predicting offsets, DNMS algorithms enable models to further reduce offset vector loss by 6.5%. To further validate the generalization of models, we tested them using a newly proposed test set, Huizhou test set, with over 7,000 manually annotated instance samples. Our algorithms and dataset will be available at https://github.com/likaiucas/OBM.