# Now you see it, Now you don't: Damage Label Agreement in Drone & Satellite Post-Disaster Imagery

arXiv: https://arxiv.org/abs/2505.08117 · HTML: https://mlsanity.com/paper/2505.08117/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 9 of 9)

- [FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding](https://mlsanity.com/paper/2012.02951/) (2020)
- [BRIGHT: A globally distributed multimodal building damage assessment dataset with very-high-resolution for all-weather disaster response](https://mlsanity.com/paper/2501.06019/) (2025)
- [RescueNet: A High Resolution UAV Semantic Segmentation Benchmark Dataset for Natural Disaster Damage Assessment](https://mlsanity.com/paper/2202.12361/) (2022)
- [MSNet: A Multilevel Instance Segmentation Network for Natural Disaster Damage Assessment in Aerial Videos](https://mlsanity.com/paper/2006.16479/) (2020)
- [Benchmark Dataset for Automatic Damaged Building Detection from Post-Hurricane Remotely Sensed Imagery](https://mlsanity.com/paper/1812.05581/) (2018)
- [CRASAR-U-DROIDs: A Large Scale Benchmark Dataset for Building Alignment and Damage Assessment in Georectified sUAS Imagery](https://mlsanity.com/paper/2407.17673/) (2024)
- [Rapid building damage assessment workflow: An implementation for the 2023 Rolling Fork, Mississippi tornado event](https://mlsanity.com/paper/2306.12589/) (2023)
- [Quantitative Data Analysis: CRASAR Small Unmanned Aerial Systems at Hurricane Ian](https://mlsanity.com/paper/2308.14577/) (2023)
- [Large Scale Organization and Inference of an Imagery Dataset for Public Safety](https://mlsanity.com/paper/1908.09006/) (2019)

## Built on by (most cited first, 2 of 2)

- [Deploying Rapid Damage Assessments from sUAS Imagery for Disaster Response](https://mlsanity.com/paper/2511.03132/) (2025)
- [Looks Can be Deceiving: Annotator and Reviewer Performance Across Imagery Sources in Crowd-Sourced Aerial Damage Assessment](https://mlsanity.com/paper/2608.14942/) (2026)

## Abstract (arXiv)

This paper audits damage labels derived from coincident satellite and drone aerial imagery for 15,814 buildings across Hurricanes Ian, Michael, and Harvey, finding 29.02% label disagreement and significantly different distributions between the two sources, which presents risks and potential harms during the deployment of machine learning damage assessment systems. Currently, there is no known study of label agreement between drone and satellite imagery for building damage assessment. The only prior work that could be used to infer if such imagery-derived labels agree is limited by differing damage label schemas, misaligned building locations, and low data quantities. This work overcomes these limitations by comparing damage labels using the same damage label schemas and building locations from three hurricanes, with the 15,814 buildings representing 19.05 times more buildings considered than the most relevant prior work. The analysis finds satellite-derived labels significantly under-report damage by at least 20.43% compared to drone-derived labels (p<1.2x10^-117), and satellite- and drone-derived labels represent significantly different distributions (p<5.1x10^-175). This indicates that computer vision and machine learning (CV/ML) models trained on at least one of these distributions will misrepresent actual conditions, as the differing satellite and drone-derived distributions cannot simultaneously represent the distribution of actual conditions in a scene. This potential misrepresentation poses ethical risks and potential societal harm if not managed. To reduce the risk of future societal harms, this paper offers four recommendations to improve reliability and transparency to decisio-makers when deploying CV/ML damage assessment systems in practice