# Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Series

arXiv: https://arxiv.org/abs/2112.07558 · HTML: https://mlsanity.com/paper/2112.07558/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 60 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Lightweight Temporal Self-Attention for Classifying Satellite Image Time Series](https://mlsanity.com/paper/2007.00586/) (2020)
- [Multi-temporal Sentinel-1 and -2 Data Fusion for Optical Image Simulation](https://mlsanity.com/paper/1807.09954/) (2018)

## Built on by (most cited first, 30 of 49)

- [SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery](https://mlsanity.com/paper/2312.10115/) (2023)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities](https://mlsanity.com/paper/2412.14123/) (2024)
- [PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models](https://mlsanity.com/paper/2412.04204/) (2024)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensing](https://mlsanity.com/paper/2507.13812/) (2025)
- [Fields of The World: A Machine Learning Benchmark Dataset For Global Agricultural Field Boundary Segmentation](https://mlsanity.com/paper/2409.16252/) (2024)
- [GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI](https://mlsanity.com/paper/2511.15658/) (2025)
- [OpenForest: A data catalogue for machine learning in forest monitoring](https://mlsanity.com/paper/2311.00277/) (2023)
- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [Planted: a dataset for planted forest identification from multi-satellite time series](https://mlsanity.com/paper/2406.18554/) (2024)
- [SICKLE: A Multi-Sensor Satellite Imagery Dataset Annotated with Multiple Key Cropping Parameters](https://mlsanity.com/paper/2312.00069/) (2023)
- [Adaptive Fusion of Multi-view Remote Sensing data for Optimal Sub-field Crop Yield Prediction](https://mlsanity.com/paper/2401.11844/) (2024)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation](https://mlsanity.com/paper/2405.01217/) (2024)
- [FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping](https://mlsanity.com/paper/2506.07080/) (2025)
- [Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications](https://mlsanity.com/paper/2301.01200/) (2022)
- [S4: Self-Supervised Sensing Across the Spectrum](https://mlsanity.com/paper/2405.01656/) (2024)
- [Increasing the Robustness of Model Predictions to Missing Sensors in Earth Observation](https://mlsanity.com/paper/2407.15512/) (2024)
- [On the Generalizability of Foundation Models for Crop Type Mapping](https://mlsanity.com/paper/2409.09451/) (2024)
- [TESSERA v2: Scaling Pixel-wise Earth Foundation Models](https://mlsanity.com/paper/2607.03949/) (2026)
- [A systematic review of the use of Deep Learning in Satellite Imagery for Agriculture](https://mlsanity.com/paper/2210.01272/) (2022)
- [Exploring Self-Attention for Crop-type Classification Explainability](https://mlsanity.com/paper/2210.13167/) (2022)
- [Impact Assessment of Missing Data in Model Predictions for Earth Observation Applications](https://mlsanity.com/paper/2403.14297/) (2024)
- [A Comprehensive Review of Agricultural Parcel and Boundary Delineation from Remote Sensing Images: Recent Progress and Future Perspectives](https://mlsanity.com/paper/2508.14558/) (2025)
- [Towards On-Board Panoptic Segmentation of Multispectral Satellite Images](https://mlsanity.com/paper/2204.01952/) (2022)
- [Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach](https://mlsanity.com/paper/2501.01132/) (2025)
- [Towards Scalable Foundation Model for Multi-modal and Hyperspectral Geospatial Data](https://mlsanity.com/paper/2503.12843/) (2025)
- [An Analysis of Temporal Dropout in Earth Observation Time Series for Regression Tasks](https://mlsanity.com/paper/2504.06915/) (2025)
- [CoDEx: Combining Domain Expertise for Spatial Generalization in Satellite Image Analysis](https://mlsanity.com/paper/2504.19737/) (2025)

## Abstract (arXiv)

Optical and radar satellite time series are synergetic: optical images contain rich spectral information, while C-band radar captures useful geometrical information and is immune to cloud cover. Motivated by the recent success of temporal attention-based methods across multiple crop mapping tasks, we propose to investigate how these models can be adapted to operate on several modalities. We implement and evaluate multiple fusion schemes, including a novel approach and simple adjustments to the training procedure, significantly improving performance and efficiency with little added complexity. We show that most fusion schemes have advantages and drawbacks, making them relevant for specific settings. We then evaluate the benefit of multimodality across several tasks: parcel classification, pixel-based segmentation, and panoptic parcel segmentation. We show that by leveraging both optical and radar time series, multimodal temporal attention-based models can outmatch single-modality models in terms of performance and resilience to cloud cover. To conduct these experiments, we augment the PASTIS dataset with spatially aligned radar image time series. The resulting dataset, PASTIS-R, constitutes the first large-scale, multimodal, and open-access satellite time series dataset with semantic and instance annotations.