# Semantics from Space: Satellite-Guided Thermal Semantic Segmentation Annotation for Aerial Field Robots

arXiv: https://arxiv.org/abs/2403.14056 · HTML: https://mlsanity.com/paper/2403.14056/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Self-supervised Augmentation Consistency for Adapting Semantic Segmentation](https://mlsanity.com/paper/2105.00097/) (2021)
- [Towards Geospatial Foundation Models via Continual Pretraining](https://mlsanity.com/paper/2302.04476/) (2023)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping](https://mlsanity.com/paper/2210.10732/) (2022)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation](https://mlsanity.com/paper/1907.10303/) (2019)
- [Boundary Loss for Remote Sensing Imagery Semantic Segmentation](https://mlsanity.com/paper/1905.07852/) (2019)
- [Caltech Aerial RGB-Thermal Dataset in the Wild](https://mlsanity.com/paper/2403.08997/) (2024)
- [Semantics through Time: Semi-supervised Segmentation of Aerial Videos with Iterative Label Propagation](https://mlsanity.com/paper/2010.01910/) (2020)
- [Building an Aerial-Ground Robotics System for Precision Farming: An Adaptable Solution](https://mlsanity.com/paper/1911.03098/) (2019)
- [Unsupervised RGB-to-Thermal Domain Adaptation via Multi-Domain Attention Network](https://mlsanity.com/paper/2210.04367/) (2022)
- [Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning](https://mlsanity.com/paper/2107.03919/) (2021)
- [MassMIND: Massachusetts Maritime INfrared Dataset](https://mlsanity.com/paper/2209.04097/) (2022)
- [A Multispectral Automated Transfer Technique (MATT) for machine-driven image labeling utilizing the Segment Anything Model (SAM)](https://mlsanity.com/paper/2402.11413/) (2024)
- [Online Self-Supervised Thermal Water Segmentation for Aerial Vehicles](https://mlsanity.com/paper/2307.09027/) (2023)
- [WIT-UAS: A Wildland-fire Infrared Thermal Dataset to Detect Crew Assets From Aerial Views](https://mlsanity.com/paper/2312.09159/) (2023)
- [Weakly Supervised Caveline Detection For AUV Navigation Inside Underwater Caves](https://mlsanity.com/paper/2303.03670/) (2023)

## Built on by (most cited first, 2 of 2)

- [SLTNet: Efficient Event-based Semantic Segmentation with Spike-driven Lightweight Transformer-based Networks](https://mlsanity.com/paper/2412.12843/) (2024)
- [SAM2-ELNet: Label Enhancement and Automatic Annotation for Remote Sensing Segmentation](https://mlsanity.com/paper/2503.12404/) (2025)

## Abstract (arXiv)

We present a new method to automatically generate semantic segmentation annotations for thermal imagery captured from an aerial vehicle by utilizing satellite-derived data products alongside onboard global positioning and attitude estimates. This new capability overcomes the challenge of developing thermal semantic perception algorithms for field robots due to the lack of annotated thermal field datasets and the time and costs of manual annotation, enabling precise and rapid annotation of thermal data from field collection efforts at a massively-parallelizable scale. By incorporating a thermal-conditioned refinement step with visual foundation models, our approach can produce highly-precise semantic segmentation labels using low-resolution satellite land cover data for little-to-no cost. It achieves 98.5% of the performance from using costly high-resolution options and demonstrates between 70-160% improvement over popular zero-shot semantic segmentation methods based on large vision-language models currently used for generating annotations for RGB imagery. Code will be available at: https://github.com/connorlee77/aerial-auto-segment.