# Using Social Media Images for Building Function Classification

arXiv: https://arxiv.org/abs/2202.07315 · HTML: https://mlsanity.com/paper/2202.07315/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching](https://mlsanity.com/paper/2005.03860/) (2020)
- [Bridging the Domain Gap for Ground-to-Aerial Image Matching](https://mlsanity.com/paper/1904.11045/) (2019)
- [Localizing and Orienting Street Views Using Overhead Imagery](https://mlsanity.com/paper/1608.00161/) (2016)
- [Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US](https://mlsanity.com/paper/1702.06683/) (2017)
- [Building Instance Classification Using Street View Images](https://mlsanity.com/paper/1802.09026/) (2018)
- [Learning a Dynamic Map of Visual Appearance](https://mlsanity.com/paper/2012.14885/) (2020)
- [Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images](https://mlsanity.com/paper/1802.02668/) (2018)
- [Changes in Twitter geolocations: Insights and suggestions for future usage](https://mlsanity.com/paper/2108.12251/) (2021)
- [Geocoding of trees from street addresses and street-level images](https://mlsanity.com/paper/2002.01708/) (2020)
- [Race, Religion and the City: Twitter Word Frequency Patterns Reveal Dominant Demographic Dimensions in the United States](https://mlsanity.com/paper/1605.02951/) (2016)

## Built on by (most cited first, 5 of 5)

- [Geo-Information Harvesting from Social Media Data](https://mlsanity.com/paper/2211.00543/) (2022)
- [Predicting building types and functions at transnational scale](https://mlsanity.com/paper/2409.09692/) (2024)
- [Towards Large-scale Building Attribute Mapping using Crowdsourced Images: Scene Text Recognition on Flickr and Problems to be Solved](https://mlsanity.com/paper/2309.08042/) (2023)
- [CMAB: A First National-Scale Multi-Attribute Building Dataset in China Derived from Open Source Data and GeoAI](https://mlsanity.com/paper/2408.05891/) (2024)
- [Fine-Grained Building Function Recognition from Street-View Images via Geometry-Aware Semi-Supervised Learning](https://mlsanity.com/paper/2408.09460/) (2024)

## Abstract (arXiv)

Urban land use on a building instance level is crucial geo-information for many applications, yet difficult to obtain. An intuitive approach to close this gap is predicting building functions from ground level imagery. Social media image platforms contain billions of images, with a large variety of motifs including but not limited to street perspectives. To cope with this issue this study proposes a filtering pipeline to yield high quality, ground level imagery from large social media image datasets. The pipeline ensures that all resulting images have full and valid geotags with a compass direction to relate image content and spatial objects from maps. We analyze our method on a culturally diverse social media dataset from Flickr with more than 28 million images from 42 cities around the world. The obtained dataset is then evaluated in a context of 3-classes building function classification task. The three building classes that are considered in this study are: commercial, residential, and other. Fine-tuned state-of-the-art architectures yield F1-scores of up to 0.51 on the filtered images. Our analysis shows that the performance is highly limited by the quality of the labels obtained from OpenStreetMap, as the metrics increase by 0.2 if only human validated labels are considered. Therefore, we consider these labels to be weak and publish the resulting images from our pipeline together with the buildings they are showing as a weakly labeled dataset.