# Multi-Modal Vision Transformers for Crop Mapping from Satellite Image Time Series

arXiv: https://arxiv.org/abs/2406.16513 · HTML: https://mlsanity.com/paper/2406.16513/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Lightweight, Pre-trained Transformers for Remote Sensing Timeseries](https://mlsanity.com/paper/2304.14065/) (2023)
- [Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series](https://mlsanity.com/paper/1811.10166/) (2018)
- [ViTs for SITS: Vision Transformers for Satellite Image Time Series](https://mlsanity.com/paper/2301.04944/) (2023)
- [Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Series](https://mlsanity.com/paper/2112.07558/) (2021)
- [Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification](https://mlsanity.com/paper/2306.01523/) (2023)

## Built on by (most cited first, 1 of 1)

- [Increasing the Robustness of Model Predictions to Missing Sensors in Earth Observation](https://mlsanity.com/paper/2407.15512/) (2024)

## Abstract (arXiv)

Using images acquired by different satellite sensors has shown to improve classification performance in the framework of crop mapping from satellite image time series (SITS). Existing state-of-the-art architectures use self-attention mechanisms to process the temporal dimension and convolutions for the spatial dimension of SITS. Motivated by the success of purely attention-based architectures in crop mapping from single-modal SITS, we introduce several multi-modal multi-temporal transformer-based architectures. Specifically, we investigate the effectiveness of Early Fusion, Cross Attention Fusion and Synchronized Class Token Fusion within the Temporo-Spatial Vision Transformer (TSViT). Experimental results demonstrate significant improvements over state-of-the-art architectures with both convolutional and self-attention components.