# Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery

arXiv: https://arxiv.org/abs/2605.26381 · HTML: https://mlsanity.com/paper/2605.26381/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Vision Foundation Models in Remote Sensing: A Survey](https://mlsanity.com/paper/2408.03464/) (2024)
- [Building Damage Detection in Satellite Imagery Using Convolutional Neural Networks](https://mlsanity.com/paper/1910.06444/) (2019)
- [Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution](https://mlsanity.com/paper/1905.01752/) (2019)
- [Large-scale Building Damage Assessment using a Novel Hierarchical Transformer Architecture on Satellite Images](https://mlsanity.com/paper/2208.02205/) (2022)
- [Building extraction with vision transformer](https://mlsanity.com/paper/2111.15637/) (2021)
- [Semi-supervised Learning from Street-View Images and OpenStreetMap for Automatic Building Height Estimation](https://mlsanity.com/paper/2307.02574/) (2023)
- [Improving Facade Parsing with Vision Transformers and Line Integration](https://mlsanity.com/paper/2309.15523/) (2023)

## Abstract (arXiv)

We present a multi-modal classification framework that fuses satellite and street-level imagery through a Perceiver IO architecture operating on spatial patch tokens from a shared DINOv2 backbone. The design naturally handles a variable number of street-level views per building without padding or fixed-size pooling, and jointly predicts multi-label roof element and roof material classes. We construct a large-scale dataset of 32,135 buildings (61,672 segments) spanning ten countries, pairing satellite images with up to eight street-level views per segment and evaluating four masking strategies for isolating the target building. We propose an RGB-M masking strategy that appends the building footprint mask as a fourth input channel, providing a soft spatial prior that outperforms hard cropping across both modalities. The Perceiver IO fusion model improves over all other fusion strategies and yields substantial per-class gains for attributes visible from street level (e.g., +11.3 AP for slate, +1.3 AP for dormers), though the satellite-only baseline retains a slight advantage in macro-averaged mAP for classes that are predominantly visible from above. These results establish a scalable, flexible architecture for multi-modal building inspection that can accommodate heterogeneous inputs and multiple output tasks.