# Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

arXiv: https://arxiv.org/abs/2306.01523 · HTML: https://mlsanity.com/paper/2306.01523/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval](https://mlsanity.com/paper/2105.07921/) (2021)
- [Self-supervised Vision Transformers for Joint SAR-optical Representation Learning](https://mlsanity.com/paper/2204.05381/) (2022)

## Built on by (most cited first, 4 of 4)

- [FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients](https://mlsanity.com/paper/2401.02433/) (2023)
- [Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification](https://mlsanity.com/paper/2509.15553/) (2025)
- [Multi-Modal Vision Transformers for Crop Mapping from Satellite Image Time Series](https://mlsanity.com/paper/2406.16513/) (2024)
- [A Multi-Modal Federated Learning Framework for Remote Sensing Image Classification](https://mlsanity.com/paper/2503.10262/) (2025)

## Abstract (arXiv)

In this paper, we introduce a novel Synchronized Class Token Fusion (SCT Fusion) architecture in the framework of multi-modal multi-label classification (MLC) of remote sensing (RS) images. The proposed architecture leverages modality-specific attention-based transformer encoders to process varying input modalities, while exchanging information across modalities by synchronizing the special class tokens after each transformer encoder block. The synchronization involves fusing the class tokens with a trainable fusion transformation, resulting in a synchronized class token that contains information from all modalities. As the fusion transformation is trainable, it allows to reach an accurate representation of the shared features among different modalities. Experimental results show the effectiveness of the proposed architecture over single-modality architectures and an early fusion multi-modal architecture when evaluated on a multi-modal MLC dataset. The code of the proposed architecture is publicly available at https://git.tu-berlin.de/rsim/sct-fusion.