# Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation

arXiv: https://arxiv.org/abs/2606.25128 · HTML: https://mlsanity.com/paper/2606.25128/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [The Role of ImageNet Classes in Fr'echet Inception Distance](https://mlsanity.com/paper/2203.06026/) (2022)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [EarthSynth: Generating Informative Earth Observation with Diffusion Models](https://mlsanity.com/paper/2505.12108/) (2025)
- [Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation](https://mlsanity.com/paper/2412.03178/) (2024)
- [Data Augmentation in Earth Observation: A Diffusion Model Approach](https://mlsanity.com/paper/2406.06218/) (2024)
- [When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators](https://mlsanity.com/paper/2602.19946/) (2026)
- [Grounding Synthetic Data Generation With Vision and Language Models](https://mlsanity.com/paper/2603.09625/) (2026)
- [Evaluating Generative Models via One-Dimensional Code Distributions](https://mlsanity.com/paper/2603.08064/) (2026)
- [Survey on Remote Sensing Scene Classification: From Traditional Methods to Large Generative AI Models](https://mlsanity.com/paper/2603.26751/) (2026)
- [BELDE: Building a Large-scale Earth-observation Land-cover Dataset for Europe](https://mlsanity.com/paper/2606.20909/) (2026)

## Abstract (arXiv)

Volume and quality of datasets are crucial for deep learning model training, yet they are often constrained by availability and data acquisition costs. Synthetic data augmentation can extend existing datasets with realistic images, and the quality of these images is generally assessed through fidelity metrics such as FID, KID, IS, LPIPS and SSIM that measure structural or distributional similarity. However, such metrics, including the widely used FID, focus on visual fidelity without reflecting downstream utility, and can diverge from human perception under perturbations that are imperceptible to human observers. In this work, we systematically evaluate Earth observation datasets alongside synthetic counterparts generated by deep generative models, comparing automatic metrics against human perception and downstream tasks. Our results reveal a stark misalignment: semantics-preserving perturbations such as rotation drastically alter metric scores while leaving human recognition unaffected, and synthetic samples that score poorly on automatic metrics achieve comparable or higher perceived realism, and can improve downstream performance when combined with real data. By benchmarking semantic segmentation models trained on mixed real-synthetic datasets, we demonstrate that quality metrics rooted in ImageNet-pretrained feature spaces are unreliable indicators for geospatial data. Our findings underscore that automatic quality evaluation of synthetic datasets should be grounded in downstream task performance and human evaluation.