# Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

arXiv: https://arxiv.org/abs/2607.10841 · HTML: https://mlsanity.com/paper/2607.10841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 12 Jul 2026.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Probabilistic End-to-end Noise Correction for Learning with Noisy Labels](https://mlsanity.com/paper/1903.07788/) (2019)
- [SpaceNet: A Remote Sensing Dataset and Challenge Series](https://mlsanity.com/paper/1807.01232/) (2018)
- [Adaptive Early-Learning Correction for Segmentation from Noisy Annotations](https://mlsanity.com/paper/2110.03740/) (2021)
- [Continental-Scale Building Detection from High Resolution Satellite Imagery](https://mlsanity.com/paper/2107.12283/) (2021)
- [Pick-and-Learn: Automatic Quality Evaluation for Noisy-Labeled Image Segmentation](https://mlsanity.com/paper/1907.11835/) (2019)
- [Learning to Segment from Noisy Annotations: A Spatial Correction Approach](https://mlsanity.com/paper/2308.02498/) (2023)
- [Noisy Supervision for Correcting Misaligned Cadaster Maps Without Perfect Ground Truth Data](https://mlsanity.com/paper/1903.06529/) (2019)
- [Learning to segment from misaligned and partial labels](https://mlsanity.com/paper/2005.13180/) (2020)
- [Map-Repair: Deep Cadastre Maps Alignment and Temporal Inconsistencies Fix in Satellite Images](https://mlsanity.com/paper/2007.12470/) (2020)
- [AutoCorrect: Deep Inductive Alignment of Noisy Geometric Annotations](https://mlsanity.com/paper/1908.05263/) (2019)
- [DragOSM: Extract Building Roofs and Footprints from Aerial Images by Aligning Historical Labels](https://mlsanity.com/paper/2509.17951/) (2025)

## Abstract (arXiv)

Supervised learning for image segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment. Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network. We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time. Code and derived datasets are available at https://github.com/venkanna37/align-and-segment