# Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models

arXiv: https://arxiv.org/abs/2312.12479 · HTML: https://mlsanity.com/paper/2312.12479/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [Learning Visual Representations with Caption Annotations](https://mlsanity.com/paper/2008.01392/) (2020)
- [Context-aware Feature Generation for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2008.06893/) (2020)
- [Open Vocabulary Scene Parsing](https://mlsanity.com/paper/1703.08769/) (2017)

## Built on by (most cited first, 4 of 4)

- [Post-Hurricane Debris Segmentation Using Fine-Tuned Foundational Vision Models](https://mlsanity.com/paper/2504.12542/) (2025)
- [Open-SAT: LLM-Guided Query Embedding Refinement for Open-Vocabulary Object Retrieval in Satellite Imagery](https://mlsanity.com/paper/2605.05344/) (2026)
- [GATA2Floor: Graph attention for floor counting in street-view facades](https://mlsanity.com/paper/2605.11863/) (2026)
- [R4Tun: LLM-guided adaptive segmental tunnel lining segmentation in point clouds](https://mlsanity.com/paper/2609.11360/) (2026)

## Abstract (arXiv)

Existing building recognition methods, exemplified by BRAILS, utilize supervised learning to extract information from satellite and street-view images for classification and segmentation. However, each task module requires human-annotated data, hindering the scalability and robustness to regional variations and annotation imbalances. In response, we propose a new zero-shot workflow for building attribute extraction that utilizes large-scale vision and language models to mitigate reliance on external annotations. The proposed workflow contains two key components: image-level captioning and segment-level captioning for the building images based on the vocabularies pertinent to structural and civil engineering. These two components generate descriptive captions by computing feature representations of the image and the vocabularies, and facilitating a semantic match between the visual and textual representations. Consequently, our framework offers a promising avenue to enhance AI-driven captioning for building attribute extraction in the structural and civil engineering domains, ultimately reducing reliance on human annotations while bolstering performance and adaptability.