# Cross-attention Spatio-temporal Context Transformer for Semantic Segmentation of Historical Maps

arXiv: https://arxiv.org/abs/2310.12616 · HTML: https://mlsanity.com/paper/2310.12616/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Attention to Scale: Scale-aware Semantic Image Segmentation](https://mlsanity.com/paper/1511.03339/) (2015)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Semantic Video Segmentation by Gated Recurrent Flow Propagation](https://mlsanity.com/paper/1612.08871/) (2016)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Efficient Semantic Video Segmentation with Per-frame Inference](https://mlsanity.com/paper/2002.11433/) (2020)
- [Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2108.11819/) (2021)
- [Spatial Pyramid Based Graph Reasoning for Semantic Segmentation](https://mlsanity.com/paper/2003.10211/) (2020)
- [Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images](https://mlsanity.com/paper/2106.15754/) (2021)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Video Scene Parsing with Predictive Feature Learning](https://mlsanity.com/paper/1612.00119/) (2016)
- [Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture](https://mlsanity.com/paper/1909.10726/) (2019)
- [ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation](https://mlsanity.com/paper/1511.07053/) (2015)
- [Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing Imagery](https://mlsanity.com/paper/1811.02471/) (2018)
- [Synthetic Map Generation to Provide Unlimited Training Data for Historical Map Text Detection](https://mlsanity.com/paper/2112.06104/) (2021)
- [Learning to Predict Context-adaptive Convolution for Semantic Segmentation](https://mlsanity.com/paper/2004.08222/) (2020)

## Built on by (most cited first, 8 of 8)

- [DIGMAPPER: A Modular System for Automated Geologic Map Digitization](https://mlsanity.com/paper/2506.16006/) (2025)
- [Generative AI in Map-Making: A Technical Exploration and Its Implications for Cartographers](https://mlsanity.com/paper/2508.18959/) (2025)
- [Detecting Legend Items on Historical Maps Using GPT-4o with In-Context Learning](https://mlsanity.com/paper/2510.08385/) (2025)
- [Encoding Agent Trajectories as Representations with Sequence Transformers](https://mlsanity.com/paper/2410.09204/) (2024)
- [Automatic Uncertainty-Aware Synthetic Data Bootstrapping for Historical Map Segmentation](https://mlsanity.com/paper/2511.15875/) (2025)
- [Deep learning enables urban change profiling through alignment of historical maps](https://mlsanity.com/paper/2602.02154/) (2026)
- [Generalizable Multiscale Segmentation of Heterogeneous Map Collections](https://mlsanity.com/paper/2603.05037/) (2026)
- [Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data](https://mlsanity.com/paper/2609.38165/) (2026)

## Abstract (arXiv)

Historical maps provide useful spatio-temporal information on the Earth's surface before modern earth observation techniques came into being. To extract information from maps, neural networks, which gain wide popularity in recent years, have replaced hand-crafted map processing methods and tedious manual labor. However, aleatoric uncertainty, known as data-dependent uncertainty, inherent in the drawing/scanning/fading defects of the original map sheets and inadequate contexts when cropping maps into small tiles considering the memory limits of the training process, challenges the model to make correct predictions. As aleatoric uncertainty cannot be reduced even with more training data collected, we argue that complementary spatio-temporal contexts can be helpful. To achieve this, we propose a U-Net-based network that fuses spatio-temporal features with cross-attention transformers (U-SpaTem), aggregating information at a larger spatial range as well as through a temporal sequence of images. Our model achieves a better performance than other state-or-art models that use either temporal or spatial contexts. Compared with pure vision transformers, our model is more lightweight and effective. To the best of our knowledge, leveraging both spatial and temporal contexts have been rarely explored before in the segmentation task. Even though our application is on segmenting historical maps, we believe that the method can be transferred into other fields with similar problems like temporal sequences of satellite images. Our code is freely accessible at https://github.com/chenyizi086/wu.2023.sigspatial.git.